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PRINCIPLES AND PROCEDURES OF MULTIPLE MATRIX SAMPLING 
David M. Shoemaker 



ABSTRACT 



Multiple matrix sampling is a psychometric procedure in which a 
set of test- items is subdivided randomly into subtests of items with 
each subtest administered to different subgroups of examinees selected 
at random from the examinee population. Although each examinee rec 
only a proportion of the complete set of items, the statistical model 
employed permits the researcher to estimate the mean, variance and 
frequency distribution of test scores which would have been obtained 
by testing all examinees on all items. Contained herein is a detailed 
description of multiple matrix sampling. The topics covered range from 
an introductory discussion to the listing with expanded writeup of the 
computer program used to analyze the data. Throughout this Report an 
attempt has been made to keep the practitioner clearly in mind. 
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PRINCIPLES AND PROCEDURES OF MULTIPLE MATRIX SAMPLING 
David M. Shoemaker 



I 



Intro duction 



Multiple matrix sampling or, more popularly, item-examinee sampling, 
is a psychometric procedure whose time has come. It is the Zeitgeist. 
Descriptions of multiple matrix sampling procedures and explorations into 
areas of application are scattered over a multitude of technical journa s. 
There is no single book or article which describes, studies, and unities 
all of this material. Yet there is a need for such a document both as a 
reference source and as a textbook. 

Although statisticians have dealt for several decades with incomplete 
data problems in the: design of experiments and data analysis, the psycho 
metrician responsible primarily for the derivation of statistical proce- 
dures in mul' iple matrix sampling and the application of such procedures 
to problems in psychology and education is Frederic M. Lord. Lord and 
Novick discuss multiple matrix sampling in Chapter 11 in Statistical 
theories of mental test scores but the chapter does not encompass the 
degree of detail and depth of explanation required by the majority of 
educational research practitioners who desire to implement this research 
procedure in a particular investigation. This Technical Report has 
been designed to remedy this situation. 

Throughout this Report an attempt has been made to keep the practi- 
tioner clearly in mind. The emphasis is on the why , when , and how to 
use multiple matrix sampling. The topics covered range from an intro- 
duction to multiple matrix sampling to the listing with expanded writeup 
of the computer program used to analyze the data. All discussions and 
guidelines contained in the mc.'ograph reflect theoretical and empirica 
results reported in the literature as well as personal experiences of the 
author in implementing multiple matrix sampling in a variety of applied 
situations . 
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Characteristics, Advantages, And Applications 
Of Multiple Matrix Sampling 



The majority of contemporary psychometric procedures reflect stron 
gly the original impetus of the psychometric movement, that is, the 
measurement of individual differences [ Historically, individual differ- 
ences have been investigated, and appropriately so, using the matched- 
items model in which a single set of test items is administered in a 
standardized procedure to all, or a sample, of the examinee population 
under consideration. One exemplar of such methodology is the anthropo 
metric laboratory of Sir Francis Galton established at the Internationa 
Health Exhibition in England in 1884. Galton measured individuals rang- 
ing in age from five to eighty on such dimensions as standing height, 
sitting height, arm span, weight, breathing capacity and strength of 
pull "to supply information on the methods, practice, and uses of human 
measurement." Understandably so and undoubtedly for lack of a reasonable 
alternative, procedures appropriate for the assessment of individual 
differences have been transferred completely to investigations concerned 
primarily with the measurement of group differences. An example of a 
research design emphasizing the assessment of group differences is found 
in an investigation which contrasts treatment effects through adminis- 
tering each treatment to a group of examinees selected randomly from the 
examinee population. Given treatments A, B, and C, for example, the 
researcher is interested primarily in the behavior of .group. A as contras- 
ted with group B as contrasted with group C. Differences among individual 
examinees are of little concern. The point to be made is simply this: 
the methodology employed successfully in the assessment of individual 
differences is neither the appropriate nor the most efficient methodology 
for group assessment. Multiple matrix sampling or, more popularly, 
item-examinee sampling, has been demonstrated theoretically and empir- 
ically to be the appropriate procedure for group assessment and a proce 
dure superior to the matched— items model. 

The matched— items model and the multiple matrix sampling model are 
contrasted readily by considering the data base which would be generated 
if the entire testable population ofi N examinees were administered the 
complete set of K test items. Such a data base is illustrated in 
Figure 2.1 and the arrangement is referred to commonly as an item- 
examinee matrix. Test items are scared dichotomously frequently and 
such is the case in Figure 2.1. For example, examinee 1 passed item 1, 
failed item 2, passed items 3 and 4, and failed item 5. Within the 
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Test Items 

1 2 3 4 5 ... 




Figure 2.1: Item-examinee matrix illustrating examinee- sampling. 
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Test Items 

1 2 3 4 5 ... 




Figure 2.2: Item-examinee matrix illustrating multiple matrix sampling. 
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framework of the item-examinee matrix, the matched-items model used in 
the assessment of individual differences is referred to as the examinee- 
sampling model because all test items are administered to a subgroup of 
examinees selected at random from the population of N examinees. By 
contrast, multiple matrix sampling involves the joint sampling of examinee 
subgroups and item sub tests as illustrated in Figure 2*2* Data rom tern 
examinee sample 1 were obtained by administering a set of items selected 
at random from the population of K test items and administering t ese 
and only these items to a subgroup of examinees selected randomly from 
the population of N examinees. Replicating this procedure produces 
item- examinee samples 2 and 3 and suggests » concomi tantly > the der vat on 
of the expression "multiple matrix sampling." Statistics obtained from 
examinee-sampling and from multiple matrix sampling are used to estimate 
parameters of the N by K item-examinee matrix. It must be remembered, 
however, that the N by K item-examinee matrix illustrated in Figures 
2 1 and 2.2 is a hypothetical matrix the parameters of which are estimated 
from the subset of data gathered in practice through examinee-sampling or 
multiple matrix sampling. 



Advantages of Multiple Matrix Sampling 

A concept important in discussing the advantages of multiple matrix 
sampling and one mentioned frequently herein is the standard error of 
estimate. Assume that two experimental procedures have been developed^ 
for measuring weight and each procedure is used to obtain in a standard- 
ized manner 1000 independent measurements of the weight of a given object. 
Hypothetical measurements so acquired have been assembled into frequency 
distributions and are given in Figure 2.3. The standard error of estimate 
associated with procedure M is the standard deviation of the 1000 values 
for the weight obtained using procedure M; the standard error of esti 
mating the weight for procedure E is determined identically. The 
difference in standard errors of estimate depicted in Figure 2.3 illus- 
trates an important advantage of multiple matrix sampling over examinee- 
sampling in group assessment. Lord and Novick (1968) have demonstrated 
algebraically that, when sub tests are constructed by sampling items 
without replacement from the K-item population, the standard error in 
estimating the group mean test score using multiple matrix sampling is 
less than the standard error obtained with examinee-sampling. Further- 
more, the minimum standard error of estimate under multiple matrix 
sampling is found by administering one item to each of K random samples 
of examinees. A conclusion such as this is of major significance because 
the parameter of primary importance in many investigations is the group 

mean test score. 



To clarify t^is point, consider how such a result could have been 
determined empirically through post mortem item-examinee sampling. In 
post mortem item-examinee sampling, an existing N by K item-examinee 
data base is taken to be the population of scores and item scores from 
item-examinee samples selected randomly from this base are used to 
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Fig-xre 2.3: Hypothetical distributions o£ weight measurements resulting 

from 1000 replications of procedure M and 10Q0 replications 
of procedure E. 
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estimate parameters of interest. Although all examinees have responded 
to all items, in post mortem item-examinee sampling t ho instigator 
acts as if individual examinees had responded only to s P ec ^ ia i ^^oxi 
The standard error of estimate under examinee-sampling could be appro 
mated, for example, by selecting at random 1000 examinee subgroups and 

"stiAg each suUup^ver K items. Data from -^“^“^SSrd 
provide an estimate of the mean score over N examinees and the sta 
deviation calculated over these 1000 estimates is the standard error of 
estimating the population mean under examinee-sampling. A single estimate 
of the mean test score under multiple matrix sampling is obtained, for 
example, by dividing randomly the set of K test items into t non 
overlapping subtests containing K/t items each and administering each 
subtest to a subgroup of examinees selected at random from the Population 
of N examinees. A single estimate of the population mean is obtained 
by pooling the t estimates obtained from each item- examinee sample. 
Replicating this procedure 1000 times provides 1000 pooled ate 

the population mean test score and hence the standard error o 
associated with the particular item-examinee sampling plan us ed. (All 
computational formulas used in multiple matrix sampling are explaine 

detail in Chapter IV.) 

The advantages of multiple matrix sampling have thus * ar ***” f °°“® ed 
on the standard error of estimating the mean test score. Important also 
is the expected value or mean of the estimates of the pop “ la ^°" 
test score over replications. In Figure 2.3, the standard error of pro ce 
dure M is less than the standard error of procedure E, however, on the 
average, the values obtained using procedure E are more acc “^ e 
those obtained using procedure M (assuming that the true weight Isjh* 
value on the abscissa indicated by the pointer). A considerat 
this prompts an examination of the mean estimate of the populaP *°" ' 
test score obtained under multiple matrix sampling. The rea “^ s ®f V 
erll empirical investigations (Johnson & Lord, 1958; Lord, 1962; Plumlee, 
1964; Stufflebeam & Cook, 1967; Shoemaker, 197.0a, 1970b) using P ° ap ™° r 
tern item-examinee sampling support the conclusion that, on the g ^ 

estimates of the mean test score are extremely accurate. ■«* 

as these are to be expected since the mean of a random sample is always 
an unbiased estimator of the population mean and estimates of the mean 
test score obtained through multiple matrix sampling are no exception.) 

Shoemaker (1970b) has demonstrated ttiat this conclusion f tion° P 
additionally, for estimates of the population standard deviation. 

In addition to the statistical advantages of multiple matrix sampling 
in estimating group achievement, there are other advantages of practical 
import: (a) The testing time per examinee is reduced under multiple matrix 

sampling. ^ This is, indeed, an important consideration “ 
sarv for testing K items per examinee is frequently difficult or P 
sible to obtain. (b) Under multiple matrix sampling, the costs of scoring 
each te'^t are reduced. (c) Multiple matrix sampling as a procedure may 
be acceded Lre readily In certain situations than the -tched-lteas^^^ 
design. In a company, for example, supervisors fearing a . . , if 
may be used against their employees may be assured more convincingly if 

' 12 



- 8 - 



each employee takes only a part of a test and different employees take 
different parts, '(d) Given a limited amount of available testing time 
per examinee, performance on a larger number of test items can be approx- 
imated through multiple matrix sampling than through a matched-items design, 
(e) With multiple matrix sampling it is possible to estimate simultane- 
ously parameters of several tests. To the examinee, the test so construc- 
ted is merely another test; however, to the test constructor, the co™nos- 
ite is a collection of several tests each having parameters estimated 
through multiple matrix sampling. 



Limitations of Multiple Matrix Sampling 

Although advantages of multiple matrix sampling are more numerous 
than limitations, the latter do exist. Estimating parameters through 
multiple matrix sampling assumes that the responses of an examinee to an 
item sample are exactly those which would have been obtained had the 
examinee responded to those items embedded in the K— item test. Although 
the data available (Sirotnik, 1970; Shoemaker, 1970c) suggest that multiple 
matrix sampling is relatively immune to a context effect, there is one 
important exception: using multiple matrix sampling to estimate parameters 

of speeded tests. In this case, an examinee’s response is not indepen- 
dent of the context of the test and multiple matrix sampling should not 
be used. 

An insidious variation of the context effect occurs when multiple 
matrix sampling is used to estimate parameters for a test which is impos- 
sible to administer in practice. For example, parameters of a 500-item 
vocabulary test designed for grade one students could be estimated readily 
through multiple matrix sampling by forming 25 subtests having 20 items 
each with each subtest administered to one class of grade one students. 
Although all stvidents could respond appropriately to the 20-item test, 
data from each subtest would be used to estimate the results which would 
have been obtained had all grade one children taken the 500-item test. 

The problem is that no individual grade one student could have tolerated 
the 500-item test. 

A potentially serious limitation of multiple matrix sampling is 
found in the logistics involved in giving different tests to different 
subgroups of examinees. If test items are administered individually, 
problems are minimal. If, however, each item requires oral instructions 
by the test administrator and different tests are to be distributed among 
the examinees in the testing room, serious problems occur. In this sit- 
uation, the examinees must be segregated and isolated according to 
subtest before administering each test. If the Instructions to each 
item are written on the test booklet, administering different tests to 
different examinees within the testing room is accomplished with rela- 
tive ease. 
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Procedural Guidelines in Multiple Matrix Sampling 



Multiple matrix sampling as a procedure involves basically three 
steps: (a) a K-item test is subdivided through random or stratified- 

random sampling into subtests each having typically the same number of 
items, (b) each subtest is administered to a group of examinees selected 
randomly from the examinee population, and (c) test parameters are esti- 
mated from subtest results. Although the procedure is described eas y> 
implementing it produces many interesting questions. For example: 
many sub tests should be formed? To how many examinees should each sub test 
be administered? Is it more appropriate to administer a few sub tests _ 
containing a large number of items or a large number of subtests contain- 
ing few items? These are only a few of the questions encountered fre- 
quently when using multiple matrix sampling. Described herein are general 
guidelines for answering these and other related questions. 



Let t denote the number of subtests, k the number of items per 
subtest and n the number of examinees to which each subtest is adminis- 
tered. A specific sampling plan is denoted by (t/k/n) . For example, 
(2/25/60), (10/5/60) and 10/20/30) are three sampling plans which could 

be used to estimate the parameters of a 50-item test. With the first 
plan, 2 subtests are formed containing 25 items each with each subtest 
administered to 60 examinees; with the second plan, 10 tests with 5 items 
each with each sub test administered to 60 examinees; and with the 
third, 10 tests with 20 items each with each sub test administered to 30 
examinees. The third plan introduces an important variable in multiple 
matrix sampling, namely, the procedure used to sample items in construct- 
ing subtests. With (2/25/60) and (10/5/60) subtests are formed by 
sampling test items without replacement from the pool of 50 items. With 
(10/20/30), items are sampled without replacement for a given subtest but 
with replacement among subtests; consequently, an individual item wi^^ 
often be included in more than one sub test, but no item will be included 
twice in the same subtest. The rule is this: if the product tk is 

less than or equal to K, the sampling of items for subtests is always 
witho ut replacement ; when tk is greater than K, the sampling of items 
is without replacement for each subtest and with replacement between 
subtests. Selecting items for two subtests using the latter sampling 
procedure is demonstrated easily with a deck of cards numbered consecu- 
tively from 1 to K: (a) the deck of K cards is shuffled thoroughly, 

(b) k cards are selected at random from the deck with the numbers on 
the cards indicating those items to be included in subtest i, (c) the k 
cards are returned to the deck, (d) the card deck is reshuffled, and 
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(e) k cards are selected at random for subtest j. Although a multitude 
of sampling plans are possible, it is generally the case that tk is 
equal to or greater than K. 

Although constructing subtests having overlapping item subsets is 
desirable in that it increases the number of observations acquired by 
the sampling plan (and, hence, decreases generally the standard error 
of estimate associated with that sampling plan) , it is of critical im 
portance that, when tk is greater than K, tk be an integer multiple 
of K, and items are sampled randomly but subject to the restriction 
that each item appear with equal frequency among subtests. With 
(10/20/30), for example, the multiple is 4 and each of the 50 items 
should appear in exactly 4 subtests. Any deviation from this procedure 
results in a marked increase in the standard error of estimate. 



An important characteristic of any sampling plan used in multiple 
matrix sampling is the number of observations acquired by that plan. 
Defining one observation as the score received by one examinee on one 
item, the number of observations acquired by a samp ling .plan is equal 
to the product tkn. For example, 3000 observations are acquired by 
t (2/25/60) and by (10/5/60) while 6000 observations are acquired by 

(10/20/30). The number of observations per sampling plan is an impor- 
tant concept in multiple matrix sampling and one mentioned frequently 
herein. 

In multiple matrix sampling, a variety of sampling plans are 
possible with the selection of a particular sampling plan being typically 
the result of both practical and statistical considerations. Determining 
the relative merits of individual sampling plans is accomplished readily 
through a consideration of the standard error of estimate for each 
parameter for each sampling plan. Shoemaker (1970a, 1970b, 1971a, 1971b) 
has determined empirically, through post mortem item-examinee sampling, 
standard errors of estimate for selected parameters as a function of 
variations in (a) the number of observations acquired by the sampling 
plan, (b) t, k, and n, (c) test reliability of the normative distribution 
of test scores, (d) the variance of item difficulty indices, and (e) 
degree of skewness in the normative test score distribution. The 
following are general guidelines in multiple matrix sampling resulting 
from these and other investigations (Shoemaker & Osbum, 1968; Osbum, 
1969) : 

1. The number of observations acquired by the sampling plan is an 
important variable. In general, as the number of observations 
increases, the standard error of estimating parameters decreases. 

(The major exception to this guideline occurs when guideline 4 

is not followed.) 

2. Increasing the number of examinees per subgroup is leas;, effective 
in reducing the standard error of estimate. 
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3. For normal normative distributions, increases in the number of 

items per subtest are most effective in reducing standard errors 
of estimate; for negatively-skewed distributions, increases in 
the number of subtests are most effective. 

4 When tk is greater than K, tk should be an integer multiple 
of K and items should be selected randomly but subject to the 
restriction that among subtests each item appears with equal 
frequency. 

5. In general, fewer observations are required to estimate parameters 
of a skewed normative distribution than of a normal normative 
distribution. 

6. If subtest items are being selected according to a stratified- 
random sampling plan instead of a random sampling plan, items 
should be stratified according to difficulty level and not 
according to content. 

7 As the reliability of the normative distribution of test scores 

increases, it becomes increasingly difficult to estimate parameters. 
For this reason, it is true generally that a relatively large 
number of observations is required by the sampling plan when 
estimating parameters of a distribution having high reliability. 

This is true also when the variance of item difficulty indices 
is large. 

8. If no information concerning the normative distribution of test 
scores is available, select a sampling plan having the number of 
subtests equal to the square root of the total number of test items 
(rounded to the nearest integer) with each subtest having approx- 
imately the same number of test items. 



Guidelines such as these are concerned primarily with relative 
standard errors of estimate in multiple matrix sampling. Although Lord 
and Novick (1968, equation 11.12.3) have determined algebraically the 
standard error of estimating the mean proportion correct score in 
multiple matrix sampling given nonoverlapping random samples of dichot- 
omous ly-s cored items drawn without replacement from the item population, 
the standard error of estimate for any parameter using any and all 
sampling plans may be determined easily and effectively through use 
of the simulation model for multiple matrix sampling described in detail 

in Chapter V. 




Multiple Matrix Sampling Step by Step 

Step Is Construct or select the K-item test. If possible, assemble 
the items into strata according to difficulty level. 

Step 2: Determine the limitations and restrictions which must be 

imposed upon the test administration procedure. 
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SteD 3* Select a sampling plan which is appropriate in view of 
the known characteristics of thenormative distribution, the restriction 
and limitations inherent in the test administration procedure, and 
guidelines 1 through 8. 



Step 4: Administer subtests to examinees in a standardized 

procedure. Avoid confounding sub tests with examinee subgroups, 
make every attempt to have examinee subgroups homogeneous. 



i.e. 



* 



Step 5; Compute estimates of parameters using equations 4.1 
4.4, 4.5, 4.7 and 4.9 with the computer program given in Appendix 



4.2, 

A. 
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Computational Formulas in Multiple Matrix Sampling 



Computational formulas used in multiple matrix sampling are applied 
easily in practice and are detailed and sequenced appropriately in the 
following application of the procedure. It should be noted initial y 
that all formulas assume uniform item scoring procedures; for example, 
some items cannot be scored dichotomous ly and other trichotomously . 



An Application of Multiple Matrix Sampling 



A spelling program is being designed for kindergarten students and 
th. word and rula content of this program is to be related closely to the 
reading program used by these students. Before constructing such a program 
it is necessary to determine the spelling proficiency of those students 
who have used the reading program but have not had formal spelling ins true 
tion on the related words. Although there were 78 unique words introduced 
in the particular reading program under consideration, technical consider 
ations dictated that only words having regular spellings be included in 
the word population. As a result, the original word population was reduced 
from 78 words to 50 words. The modified word population was then subdi 
vided through random sampling without replacement into su tests con a— n 
ing 10 words each. (This is one of many procedures which could have been 
used. Alternative procedures are discussed in detail in Chapter III.) 

Three kindergarten classes were selected randomly from the pool o 
classes. Students within each class were divided at ranaom into 5 groups 
and each group was assigned at random to one of the 5 sub tests. Each s 
was administered individually. All items were scored dichotomously (1 
pass, 0 = fail) with the results of each subtest given in Tables 4.1 

through 4.5 . 



Estimating Parameters From Subtest R esults 



In multiple matrix sampling, sub test results are of secondary 
interest. Of chief concern is the estimation of parameters, that is, 
the results which would have been obtained had all students been tested 
over the entire set of 50 items comprising the word population. The 
results of each sub test, however, can be used to provide estimates o 
parameters of interest. For example, from sub test 1 it is possible to 
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obtain an estimate of several parameters, i.e., M (the ™“" ce 

test score), a (the standard deviation of test scores), (the varia 
Of test scores) , M 3 (the third moment about the arithmetic mean) . U 4 (th 
fourth moment about the arithmetic mean) , a 2 l (the coefficient of rl 
bility) , gi (the index of skewness) , and g 2 (the degree of kur *° £*. 

All of thfse parameters are not independent but each can * ^uUiple 
from the results of one subtest. In multiple matrix P 

subtests are used and, hence, multiple estimates of each pa ^ a ^ r ar 
obtained. A more accurate estimate of each parameter is obtai 7 

combining or pooling the estimates obtained from each subtest. 

Although it is possible to estimate several parameters, the majority 
of investigations are interested primarily in estimating V, a » 5“ “ 21 * 

The appropriate formulas for estimating these parameters from subt 

are 



“t 



KT. 

X 



k i 



(4.1) 



a 2 

O . 
x 



n.K{(K - l)s£ - (K 



k. 

x 

k i )C v^ 



k.(k. - 1 ) (n - 1 ) 
i i x 



(4.2) 



and, 



A 

a 



21 , 



K 

K - 1 



A 

M- . 



1 - 



L 



A 2 

■‘i 



K 



A 2 
O' . 
X 



(4.3) 



where, 



K = the total number of items in the population, 
k, = the number of items in subtest i, 



n. = the number of examinees receiving subtest i, 
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1.9167 
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= the mean test score on sub test i, 

s 2 B E^(T - Tj)^/n 4 , the variance of test scores on subtest i, 
°± i 1 • 



and 



k 



i 



£ v 



i 



the sum of the k ± item variances in subtest i. 
If items are scored dichotomously , the variance 
of item j is equal to p^(l " Pj) where is 
the proportion of examinees answering item j 
correctly- 



The computational formula for S 2 was derived from an ^elated 
formula given by Sirotnik (1970) in which it was ass wed that Re number 
of examinees and number of items in the population were both finite. 
Formula 4.2 is based on the assumption that the number of examinees in 
the population is infinite and that the number of items in .he popu a- 
tion is finite - 

The results of each subtest provide an estimate of y and a and 
a pooled estimate of y and a2 is obtained by combining the 

estimates using 



sub test 



pooled 



S _»A 

£ o. 



(4.4) 



and 



a 2 

pooled 



a2 

£ 0 i a ± 



£ 0 . 



(4.5) 



where , 



0^ 



(4.6) 
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the numher~of observations obtained from sub test i. If the total number 
of examinees Enj. = N is less than 500 , S| ooled should be multiplied 



bv fN - 1)/N. Pooled estimates of y and a 2 for the word spelling project 
are given in Table 4.6. The pooled estimate of the mean test score on 
the 50-item test is 20.4287. On the basis of this result, the conclusion 
was made that kindergarten students “n spell correctly approxtoately 
40 per cent of words having regular spelling in the reading program with 
out having had any formal spelling instruction. 



Although individual estimates of the reliability of the . ^"^into a 
test could have been obtained from each subtest and then comoined into a 
single estimate, a simpler procedure for estimating a 21 is one using t e 
pooled estimates of y and o2. Specifically, 



A 

a 



21 




a 2 

Spooled 

Spooled " K 

_ 

pooled 



(4.7) 



For the word spelling test u 21 for the 50-item test was estimated^from 
4.7 to be .9479. The exact computations are given in Table 4.7 

is computed as an intermediate step in approximating the normative 
tist score distribution with a probability distribution. 



Direct Calculation of SE (y^^^-| 

A more meaningful interpretation of y pooled i s possible if SE(y pooled ) 

is known. Although SE(p pooled ) and SE(ii pooled ) may be determined for all 

sampling plans through use of the simulation model described in Chapter V, 
Lord and Novlck (1968. equation 11.12.3) have derived “ J lven 

determining the standard error of the mean proportion correct score given 
fa) items are scored dichotomously , (b) items are sampled randomly and 

without replacement from the item population, and (c) Reg tric tions 

randomly and without replacement from the examinee population. Restrictions 
(b) and 7 (c) produce item subsets and examinee subgroups which are n°n° ver " 
lapping, i.e^ , no item is found in more than one subtest and no examinee 
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Table 4.6 

Pooled Estimates Of Parameters From Subtest Results 



Subtest 


Number of 
Observations 


n 


k 


A 

M* 


£ 2 


i 


180 


18 


10 


17.7780 


158.1778 


2 


140 


14 


10 


26.4285 


53.8430 


3 


130 


13 


10 


23.4615 


230.0509 


4 


130 


13 


10 


24.6155 


265.5789 


5 


120 


12 


10 


9.5835 


169.3378 




700 


70 








A 


(180) (17 . 7780) 


+ (140) (26.4285) 4 


• ... 4 (120) (9. 5835) 


spooled 


180 + 


140 + . 


.. 4 120 






= 


20.4287 










A 2 


(1 80) (158. 1778) + (140) (53.8430) 


+ ... 4 (120(169.3378) 


a _ , “ 

poolea 


180 + 


140 + . 


.. + 120 






= 


172.5178 












N L 500 










= 


172.5178 [(70 


- l)/70] 








170.0533 
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is found in more than one subgroup. Equation 11.12.3 when modified to give 
the standard error of the mean test score is 



VAR( %ooled ) " [ tkn^ '-(K - 1)(N - l)" 1 

CK 2 N a 2 { (K - k) (n - 1) - kn(t - 1)3 + 

K a 2 t(N - n) (k - 1) - kn(t - l)i + 

y(K - £) {(K - k)(N - n) + kn(t - 1)31 > 



where K refers to the total number of test items, 

N to the total number of examinees, 
q2 to the population variance, 

G 2 to the variance of item difficulty indices, and 
P 

y to the estimate of the population mean obtained 
from multiple matrix sampling. 

In practice, a 2 and are estimated; t, k, and n are parameters 

A A ^ 

defining the sampling plan. Of course, SE(yp^ 0 i e( j' “ ^^^pooled 
No equation is given by Lord and Novick for SE ( a poo i e( ^ under multi P le 
matrix sampling. 



Approximating the Normative D istribution 



In addition to estimating individual parameters through multiple 

matrix sampling, it is possible to estimate the entire /Tbv testing U lll 
distribution of test scores which would have been obtained * ? 

students on all 50 items. The negative hypergeometric h “ 

been shown by Keats and Lord C1962) to provide a reasonably good f it f< or 
a wide variety of test score distributions when the test scoraisthe 
number of correct responses . The negative hypergeometric distribution 
is a function of the mean test score y, the variance of the test score 
o*and the total number of items in the test K. Lord (196 : 2 > Shoenak. er 

(1970) have demonstrated the negative hypergeometric distribution 
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with parameters estimated by multiple matrix sampling can be used satis- 
factorily to approximate normative distributions o nt™ -tribution is 
scores. The formula for the negative hypergeometrrc distribution is 



c(-K) (a) T 



h(T) = 



for T - 0, 1, 2, 



K 



(4.9) 



(-b) T T! 



where , 



A A 



a " ("I + 1/a 21 ) %ooled 



b = -a - 1 + K/Qf 



21 



X K] 



C - 



(a + b) [K] 



noting that, 

b CK] = b(b . 1)(b . 2 ) ... (b - K + l) 



(a) T = a(a + l)(a + 2) ... (a+T- 1) 



(a ) 0 




( 



r£ i -s T (T *■ 1) (T “ 2) ••• (2 ) (1) . 

Using estimates of y and a 2 obtained from the word spelling 
project:, the calculations necessary for approximating the normative 
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distribution on the 50-item test with the negative hypergeometric distri 
bution are illustrated in Table 4.7 with complete results given in Table 
4.9. The computations involved in estimating y and a2 and 
mating ttfe normative distribution by the negative ypergeom . 

bution are more laborious than difficult. A computer pr °®^ a "' f 

developed which performs all the necessary computations and output for 
the word spelling project is given in Tables 4.8 and 4.9. * dabadled 

writeup and listing of the computer program is given in Append 

An examination of the estimates of parameters given In Table *.8 
sueeests that individual subtests were not equally difficult, particula y 
surest 5 Although the words included in sub test 5 were selected randomly 
from the 50-word population and administered to subgroups of ex^nees^ 

selected at random from each class, the results mere y co obvious 

known fact that extreme cases do occur through random sampling, 
advantage, then, of multiple matrix sampling over any individual item 
examinee sample is that the estimates obtained in the former case are 
based on a composite and hence less subject to sampling extremities. 

Stated more precisely, the standard error associated with the pooled 
estimate of the mean test score is less than the standard error associated 
with any of the estimates of the mean obtained from subtest • 
for y and a given in Chapter V illustrate adequately the difference 

in standard errors of estimate described here. 

The relative frequencies given in Table 4.9 are actually the 
individual probabilities associated with all possible test scores. For 
example, the probability of an examinee spelling correctly 20 words out 
of 50 is .023. An equally appropriate interpretation is that 2.3 P 
cent of the examinees in the population would spell correctly 20 words. 

As should be the case, the relative frequencies in Table 4 . 9 sum to 
unity. An estimate of the number of examinees receiving eac e 
is obtained by multiplying the total number of examinees in the populati 
by the probability associated with each test score. For example, 
were 1000 students in the population of kindergarteners, 23 stu ents wou 
be expected to spell correctly 20 of the 50 words on the test. 

Although equations 4.1 and 4.2 are appropriate for all item scoring 
procedures, the negative hypergeometric distribution is used only w en 
the test score is the number of correct answers. This is, of course, 

SI case when 6 items are scored 1 = pass and 0 - fail. When items . are 
not scored dichotomous ly, the normative frequency distribution may 

approximated by a Fearson curve using the first moment about the origin 
and the second, third and fourth moments about the mean. There are 12 
curves in the family of Pearson curves and the procedure for selecti g 
the appropriate curve and making the necessary cal< culetiojs i *^38-127) 
imate the normative distribution are given by Elderton <19 38, pp. 38 127) 
by Kendall (1952, pp. 137-1A5) . lord (I960) has sugge sted th at a 
Pearson Type I curve may be an appropriate selection. t , f 

tioned however, that such a procedure is not a casual undertaking. Before 
such procedures can be vised, computational formulas for estimating y 3 an 
p must be derived. Guidelines for estimating these moments are given by 

Hooke ( 1956) . 
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Table 4.7 

Computations For Negative Hypergeometric Distribution 



= (50/(50 - 1))(1 - (20.4286 - 20.4286 2 /50)/170. 0552 ) = .9479 
a - (-1 + 1/. 9479)(20.4287) = 1.1226 
b = -1.1226 - 1 + 50/. 9479 = 50.6250 

50.6250^ 5 °^ 50. 6250(50. 6250-l)(50. 6250-2) ... (50.6250-49) 

c = 

51 7476^ 50 ^ 51. 7476(51. 7476-l)(51. 7476-2) ... (51.7476-49) 



h(0) = (.0214) 



(-50) 0 (1.1226) 0 
(-50.6250) 0 0! 



( 1 )( 1 ) 

(.0214) = .0214 

( 1 )( 1 ) 



L 



h(l) = (.0214) 



(-50) 1 (1.1226) 1 
(-50.6250)' lT 



(-50) (1 . 1226 ) 
(.0214) (_ 50>62 50)(i) 



.0237 



h (2) = (.0214) 



(-50) 2 (1.1226) 2 
7“50.6250) 2 2! 



(.0214) 



(-50) (-49) (1.1226) <2. 1226) 
(-50.6250) (-49.6250) (2) 



0214 



= .0248 
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Table 4.8 

Estimates Of Parameters For Word Spelling Project 



Sample 


Estimate 


Of Parameter 


Mean 


Variance 


1 


17.7777770 


158.1844600 


2 


26.4285710 


53.8461540 


3 


23.4615380 


230.0498600 


4 


24.6153840 


265.5769300 


5 


9.5833331 

1 


169.3392200 


POOLED 


MEAN “ 20.4285710 




POOLED 


VARIANCE = 170.0552200 
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Table 4.9 

Estimated Relative Frequency Per Test Score On The 50- Item Test Using 

The Negative Hypergeometric Distribution 



Score 



Relative 

Frequency 



Score 



Relative 

Frequency 



0 


.0213564 


1 


.0236785 


2 


.0248133 


3 


.0254953 


4 


.0259319 


5 


.0262115 


6 


.0263806 


7 


.0264667 


8 


.0264872 


9 


.0264544 


10 


.0263766 


11 


.0262602 


12 


.0261101 


13 


.0259298 


14 


.0257223 


15 


.0254900 


16 


.0252347 


17 


.0249578 


18 


.0246608 


19 


.0243444 


20 


.0240095 


21 


.0236568 


22 


.0232868 


23 


.0228997 


24 


.0224960 


25 


.0220756 



26 


.0216387 


27 


.0211852 


28 


.0207151 


29 


.0202280 


30 


.0197236 


31 


.0192016 


32 


.0186613 


33 


.0181020 


34 


.0175230 


35 


.0169233 


36 


.0163017 


37 


.0156567 


38 


.0149867 


39 


.0142896 


40 


.0135627 


41 


.0128031 


42 


.0120066 


43 


.0111683 


44 


.0102815 


45 


.0093369 


46 


.0083215 


47 


.0072158 


48 


.0059869 


49 


.0045728 


50 


.0028209 
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Computational Irregula rities 

In estimating parameters from subtests having a ^all number of 
examinees, it happens frequently that a J ‘ is equal to zero 
oTTs k s £ zero for one or more subtests. ^Although uninterpretable, 
estimates such as these should not be discarded or set equal to zero in 
computing &2 . It must be remembered that results of any subtest 

pooled 

are relatively unimportant; what is important is the accuracy of the 
pooled estimate of o? Any procedure which ignores Pa rt °f the data 
oroduces an estimate of 0 ^ which is biased, i«e«, it w P 

proach the true value even if the number of subtests was increased 
indefinitely. Sirotnik (1970) has verified empirically this conclusion. 
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Computer Simulation of Multiple Matrix Sampling 



In evaluating a particular sampling plan or contrasting the relative 
merits of several plans used in multiple matrix sampling, ^atlstics of 
primary importance are the standard error of estimate and the ™ean estimat 
for each parameter given that sampling plan. For examp e, f . 

gator were estimating parameters of a 50--item test us ng P c 

sampling, one sampling plan might be (5/20/30); another, ( I I )• 
the first sampling plan, the 50-item test is subdivided through random 
sampling without replacement within subtests and with replacement between 
sub tests into 5 subtests containing 20 items each with ^ch subtest admin- 
istered to 30 examinees; in the second plan, similar y» 
taining 10 items each with each subtest administered to 60 examinees. 

The sampling plan selected will be used only once in an investigation, 
yet, in a electing the particular plan to be used, the investigator must 
bT»ar. of the Standard error of estimate associated with each sampling 
plan under consideration. Lord and Novick (1968, equation 11.12.3) ha 
derived algebraically the standard error of estimating the mean propor- 
tion correct score given nonoverlapping random samples of dichotomous ly- 
scored items drawn without replacement from the item population. o 
comparable equation is given by them for computing the standard error of 
estimating the population standard deviation under multiple matrix sam 
pling. What is required, however, are equations for estimating sta 
errors of estimate per parameter for all potentially useful sampling p an , 
not just those plans involving nonoverlapping random samples of items 
from the item population. The computational difficulties in such a task 
are not minor; however, the results of such equations are approximated 
readily and to any desired degrea of accuracy through the computer 
simulation model described herein. The remaining sections of this chapter 
are devoted to a detailed discussion of a simulation model for multiple 
matrix sampling. The reader uninterested in such matters can bypass 
safely this chapter without a loss of continuity. However, several of 
the guidelines for multiple matrix sampling given in Chapter III are 
based on results obtained through use of this model and, it must be 
stressed, that the results obtained are only as good as the simulation 

model used. 
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Simulated Post Mortem Sampling 

The algorithm used within the model is described most appropriately 
as simulated post mortem multiple matrix sampling. In post mortem 
sampling, item-examinee samples are taken from an N by K data base 
obtained by testing N examinees over K items; in simulated post 
mortem sampling, the N by K data base is computer-generated by a 
simulation model. Generating data bases with prescribed parameters is 
essential in investigating hypotheses in multiple matrix sampling be- 
cause it is difficult, if not impossible, to locate existing data bases 
having the necessary variation in test parameters. For example, if the 
standard error of estimate were being investigated as a function of vari- 
ation in item difficulty indices for a given test reliability and test 
length, it would be difficult locating data bases with - .00, .05, 

and .08 all having CX 20 = .80 and the same test length. Such a problem 
is, however, handled easily with a simulation model. As . an overview, 
the computer program generates a data base, selects multiple item- 
examinee samples from this data base, performs all calculations necessary 
for estimating parameters, and replicates this procedure as many times 
as specified before computing the standard error of estimate and mean 
estimate per parameter over replications. The computer program is re- 
stricted to data bases having dichotomous ly-s cored items and, in multiple 
matrix sampling, to subtests having an equal number of items and examinees 



Generation of Data Bases 

In simulating multiple matrix sampling, generation of the data base 
is of primary importance. Although one procedure might be that of gener- 
ating an N by K matrix and storing it in memory, a more appropriate 
procedure is one in which the item scores on the K-item test are gener- 
ated for one and only one individual at a time. All that is stored in 
memory are the K item scores for one individual. The procedure, how- 
ever, for generating item scores must be one such that, over any number, 
of hypo thetical examinees generated , the items and test scores have 
prescribed characteristics. In this procedure, the population of examinees 
N is countably infinite. The test parameters subject to manipulation 
within the program are; (a) K, the number of items in the item popula- 
tion, (b) y , the mean test, score over examinees, (c) aS the variance of 
test scores over examinees, (d) <*20’ t * le coe ^ ficient reliability for 

the K-item test, (e) q 2 , the variance of the item difficulty indices, 

where, the difficulty index p ± for item i is the proportion of examinees 
answering correctly item i, and (f) the degree of skewness in the distri- 
bution of test scores for examinees on the K-item test. In the computer 
program, values for K, y and a2 must be specified by the user. The max- 
imum value for K is 150; y is, therefore, restricted to values 0 < y <K. 
If a 2o is specified, cr 2 is determined by the well-known relationship 
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K 



a 



20 K 



a 



2 

|j, g 

H +~K + & 



(5.1) 



derived originally by Tucker (1949). If ° 2 is specified by the user, a 2 0 
is determined consequently. Such an arrangement has been i ^°^ or ^ ed 
within the program to facilitate hypothesis testing where either c or 
aoo is to be controlled across levels of K. Of course, p V/ 
miSedonce U has been specified. The degree of in the nonwtixve 

distribution is simulated by using the lognormal or normal 

distribution functions to generate test score distributions. The ^2™^ 
mal distribution with two parameters is used to generate P°si -.JJv u! . lon 
test score distributions while the three parameter lognormal distribution 
is used for negatively-skewed distributions. The lognormal distribution 
is described in detail by Aitchison and Brown (1957) and * 
nation of simulating stochastic variates with tne lognormal distributi 
Js given by Naylor, Balintfy, Burdick and Chu (1966). The normal density 
function Z, of coirse, used to simulate normal teat score distributions. 
Density functions for the two and three parameter lognormal probability 
distributions are, respectively. 



■ 2 n * r f ln(T) -M>} 2 _ -I 

< T i ,i > c 5 -&m expC ■ 2c 2 ] 



-A- (T'| T-K-T,n.u 2 ) expC- 



{ In (T ) - |x3‘ 



-3 



2a' 



(5.2) 

(5.3) 



for T = 0 , 1, 2, ... > 

For the normal distribution, the density function is 

1 (T - H) 2 

5 “im exp c ' 2o 2 



(5.4) 
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The constants y and P 2 in 5.4 are equal, respectively, to the desired 
mean and variance in the normative distribution; however, in 5.2 and 5.3 
u and 0 2 are a function of the desired mean and variance in the norma- 
tive distribution. If the desired mean and variance of the normative 
distribution are denoted, respectively, by a and 6, y and a in 5.2 
and 5.3 are computed by 



|i = ln(a) 



ln(p 2 A* 2 + 1) 

2 



(5.5) 



and 



a 2 = In ((3 2 A* 2 + 1). (5.6) 



The appropriate derivations for 5.5 and 5.6 are given by Naylor, Balintfy, 
Burdick and Chu (1966). If z is a random normal deviate N(0,1), test 
scores T having lognormal distributions are generated by 

T = exp (y + i=l,2, ... , N (5.7) 



for positively-skewed distributions, and 

T i = K - exp(M- + oz.) ±=1,2, ... , N (5.8) 



for negatively-skewed distributions. 



For normal distributions, 



T. 

l 



= y. 



+ a z . 

l 



1 = 1 , 2 , 



N 



(5.9) 



The T scores computed in 5.7, 5.8 and 5.9 will be continuous variables. 
Because items are scored dichotomous ly , the T score must be rounded to 
the nearest integer value. The midpoint of each score interval is taken 
to be that point above which one-half of the area in that score interva 
is found. This point is found by integrating via trapezoid rule the area 
under the appropriate normal or lognormal curve. If the T score is 
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equal to or greater than the midpoint, the score is rounded up; if not, 
the score is rounded down. 

Item scores are related to test scores. Specifically, if X ±j is 

the item score for examinee i on item j, S X_ = . Also, p = T/K. If 

^2 greater than zero, Individual item difficulty indices are generated 
P 



by 



p . = p + a z . 
F i F pi 



i=l,2, 



K 



(5.10) 



where z is a random normal deviate. When is not equal to zero, the 

1 P _ 2 
distribution of p. values will be approximately normal. IZ a is equal 

1 2 . 

to zero, p^ = p for all values of i. With skewed distributions, is 

typically 0 < < .001 and, because of this, is set to zero for all 

skewed distributions generated by the simulation model. After the item 
difficulty indices have been generated within the program, deciding if an 
examinee passes or fails each item is relatively simple. Item difficulty 
indices are computed for all items generated. An examinee passes those 
items which will bring the computed item difficulty indices most closely 
to the desired item difficulty indices. For example, if the computed item 
difficulty for item i were less than the desired item difficulty for item l 
examinee j would pass item i; if the computed difficulty were 3< i ua l to or 
greater than the desired item difficulty, he "fails item i. m the 
program, the desired item difficulty indices are sorted in descending 
order. If, in following the algorithm from the first through the Kth 



item, EX. # T . , the first T - £ X^ = d items not already passed by 
i J 3 J 

examinee j are scored by the program as items answered correctly by him 



The validity of the simulation model is found in its ability to gener 
ate the desired data base. Two examples of data bases generated by the model 
are given in Tables 5.1 and 5.2. Although the discrepancies in Table 5.2 are 
minor, it should be noted that the magnitude of the discrepancies decreases 
with increases in K. 



Simulation of Multiple Matrix Samp ling 

Sub tests are constructed within the program by sampling at random items 
from the K-item population. For example, if K equals 50 and a (5/10/30) 
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Table 5.1 

Results Obtained From Simulation Model For 3000 Examinees When K - 20 
With The Normative Distribution Distributed Normally 



Parameter 


Input 


Output 


’ K 


20 


20 


M- 


10.0000 


10.0150 


2 

CT 




computed 


O' 


.8000 


.7999 


20 


p 


.5000 


.5010 


a 2 


.0800 


.0799 


P 


N 


3000 


3000 



Item Difficulty Indices . Obtained Frequency Dist 



Item 



Input Output 



Score 



Frequency 









0 


16 


1 


.987 


.987 


1 


17 


2 


.901 


.901 


2 


36 


3 


.857 


.857 


3 


47 


4 


.782 


.787 


4 


96 


5 


.750 


.750 


5 


136 


6 


.737 


.738 


6 


182 


7 


.702 


.703 


7 


225 


8 


.634 


.634 


8 


307 


9 


.565 


. 566 


9 


266 


10 


.518 


.519 


10 


328 


11 


.458 


.458 


11 


327 


12 


.442 


.442 


12 


256 


13 


.430 


.430 


13 


210 


14 


.374 


.374 


14 


191 


15 


.275 


.275 


15 


122 


16 


.208 


.209 


16 


104 


17 


.139 


.140 


17 


67 


18 


.113 


.114 


18 


38 


19 


.112 


.113 


19 


16 


20 


.026 


.026 


20 


13 




43 
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Table 5.2 



Results Obtained From Simulation Model For 3000 Examinees When K - 20 
With The Normative Distribution Negatively-Skewed (Three Parameter 



Lognormal Distribution) 



Parameter 


Input 


Output 




K 


20 


20 




M< 


17.5000 


17.6150 




a 2 

a 


.8000 


computed 

.7570 




20 


p 


.8750 


.8810 




a 2 


.0000 


.0002 




P 


N 


3000 


3000 





Item Difficulty Indices 
Item Input Output 



1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 



,875 


.930 


.875 


,908 


.875 


.888 


.875 


.876 


.875 


.876 


.875 


.p76 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.876 


.875 


.875 


.875 


.875 



Obtained Frequency Dist. 
Score Frequency 



0 


16 


1 


3 


2 


2 


3 


5 


4 


4 


5 


3 


. 6 


3 


7 


6 


8 


7 


9 


8 


10 


15 


11 


32 


12 


32 


13 


43 


14 


84 


15 


135 


16 


200 


17 


352 


18 


6 TO 


19 


885 


20 


495 
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sampling plan were used, 5 subtests would be formed by sampling without 
replacement from the 50-item pool 10 items for each subtest. If (10/10/30) 
were vised, 10 subtests would be formed containing 10 items each; however, 
the sampling plan for items requires sampling without replacement for each 
subtest but with replacement between subtests. In (10/10/30), several 
items will be common to more than one sub test. Taking (10/10/30) as an 
example, item scores on the K-item test would be generated by the program 
for 300 examinees. For subtest 1, the data from the first 30 examinees 
would be processed for only those. items included in subtest 1- An iden- 
tical procedure is followed for sub test 2 through sub test 10. The compu 
tations performed on each item-examinee sample are identical to those 
outlined in Chapter IV. If the user opts r replications of a particular 
sampling plan, r pooled estimates of each parameter will be produced an 
the standard error of estimate per parameter with that sampling plan is 
the standard deviation of the r pooled estimates for each parameter. 
Sample output for the (10/15/30) plan with 5 replications is given on 
page 42 through 49 for the normal normative distribution case. 



Uses for the Simulation Model 

It is anticipated that the computer program for simulating multiple 
matrix sampling described herein, and listed with expanded writeup in 
Appendix B, will facilitate readily a detailed examination of the relative 
merits of one or more sampling plans in multiple matrix sampling. 
multiple matrix sampling questions asked frequently are How do I do it. 
and "If I sample this way, how accurate will the estimates be? Questions 
such as these are answered easily through use of the simulation model. 

The results obtained from the program are reasonable to the degree that 
the normative distributions can be described adequately by the normal 
and lognormal probability distributions . It is commonly known that achieve- 
ment test scores are frequently normally distributed. However, the scores 
on criterion-referenced tests, i.e., end-of-program tests, are frequently 
markedly negatively-skewed and resemble closely a three parameter log- 
normal distribution. It is anticipated that the simulation model will 
prove to be an asset in test theory and test construction courses permit- 
ting the student to have a working familiarity with sampling procedures 
used in multiple matrix sampling. 
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VI 



Hypothesis Testing and Multiple Matrix Sampling 



Parameters estimated through multiple matrix sampling are integrated 
easily into a variety of hypothesis testing procedures. For example, one- 
sample and two-sample t-tests can be performed readily with estimates of 
y and obtained by multiple matrix sampling. Specifica ly» 



fc df=N-l 



A 

Spooled ^standard 






( 6 . 1 ) 



and 



t df=N 1 +N 2 -2 



A A . V 

u - H 2 ~ < ~ ^2 ) 

pooled pooled 



(N -l)cr^ +< ^ N 2" 1 ^ Cr 2 

pooled pooled 


% 


1 1 
+ 


N,+N -2 




N 1 N 2 


1 2 




- 



( 6 . 2 ) 



The t-test for the. difference between two independent means given in 
6.2 can be extended to completely randomized and factorial analysis of 
variance designs where the dependent variable is a mean test score. Al- 
though analysis of variance designs with mean scores as the dependent 
variable are found infrequently in the literature, the frequently occunng 
circumstances in which mean scores are preferable to raw scores in such 
analyses are detailed most succinctly by Peckham, Glass and Hopkins (1969). 
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Consider the design in which the relative merits of four experimental 
training programs are being contrasted through end-of-program test scores 
obtained from students participating in each procedure. Through an analy- 
sis of pretest scores given to all students, ten classes have been selec- 
ted for each training program such that, across training programs, the 
four groups of 10 classes are approximately homogeneous at the start of 
instruction. The mean achievement test score for each class is estimated 
easily through multiple matrix sampling. The statistical layout and 
sources of variation are given in Table 6.1. If an additional variable, 
such as school district, were added to the design, the statistical lay- 
out and sources of variation are modified slightly as seen in Table 6.2. 
After the measurement on the dependent variable is accomplished, compu- 
tations in the analysis of variance proceec in the usual manner. The 
novelty herein is in estimating the class mean test score through multiple 
matrix sampling. 

2 2 _ 

Testing homogeneity of variance hypotheses of the form " 

... = is accomplished for two variances by 



Spooled 

F (0 - 1,0 - 1 ) 

°2 

pooled 



(6.3) 



and for more than two variances by, for example, 



largest 

F = 

max f 2 

a 

smallest 



(6.4) 



Tables for the F statistic have beer constructed by Hartley and are 
max 

given in Winer (1962, p. 653). Another simple test for homogeneity of 
variance cleveloped by Cochran which lends itself to multiple matrix 
sampling is 
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* 2 
a 

largest 



C = 



^p. a 2 

2-j o 



(6.5) 



and the necessary tables for the C statistic are given in Winer (1962, 
p. 654). The procedures in 6.3, 6.4, and 6.5 are not the only tests 
possible, but. they are used frequently and illustrate the concept. 

The normative distribution approximated by the negative hypergeo- 
metric distribution with parameters estimated through multiple matrix 
sampling provides the basic data, for several goodness -of- fit tests. For 
example, the Kolmogorov-Smimov one-sample test (Siegel, 1956, pp.^ 47- 
52) provides a test of the hypothesis that the approximated distribution 
of scores came from a population of scores having a specified theoretical 
distribution. The test involves specifying the cumulative frequency 
distribution which would occur under the theoretical distribution and 
comparing that with the approximated cumulative frequency distribution. 
The cumulative frequency distribution is, of course, obtained readily 
after the individual frequencies have been determined by multiplying 
the number of examinees in the population by the relative frequency per 
test score approximated by the negative hypergeometric distribution. A 
simple extension of the Kolmogorov-Smirrov one-sample test is the 
Kolmogorov-Smirnov two-sample test (Siegel, 1956, pp. 127-136) which is 
concerned with the agreement between two approximated frequency distri- 
butions . 



The tests of hypotheses mentioned herein do not constitute an ex- 
haustive listing of statistical tests to which estimates of parameters 
obtained through multiple matrix sampling are applicable. The intent is 
merely that of suggesting the applicability of a novel technique to tra- 
ditional hypothesis testing procedures. It should be noted that the t- 
tests given in 6.1 and 6.2 are to be considered conservative tests of the 
hypotheses under consideration. The standard errors of estimate given 
in the denominators are those for the matched-items design and there is 
evidence (Osbum, 1967) suggesting that the corresponding standard errors 
under multiple matrix sampling will be less. In the algebraic derivation 
supporting this conclusion, Osbum was considering a form of multiple 
matrix sampling in which k items were selected at random from the pop- 
ulation of items for each examinee. 
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Table 6.1 



Statistical Layout For One-Way Analysis Of Variance Problem With The 
Dependent Variable Being A Mean Achievement Test Score Estimated Through 

Multiple Matrix Sampling 



Program 


1 2 


3 4 


A A 

M* u 

pooled pooled 

A A 

M* 2 ^ 12 

pooled pooled 

• « • • • • 

A A 

^ 10 J 1 20 - , 

pooled pooled 


A A 

M»2i ^31 

pooled pooled 

A. A 

M*22 ^32 

pooled pooled 

• • • • • • 

^30 , . *40 , . 

pooled pooled 




Source Of Variation 


Degrees Of Freedom 


Programs 


3 


Classes Within Programs 


36 


Total 


39 
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Table 6.2 

Statistical. Layout For Factorial (Two-Way) Analysis Of Variance Problem 
With The Dependent Variable Being A Mean Achievement Test Score Estimated 
Through Multiple Matrix Sampling 



District A 



District B 



Program 





A 


A 


A 


A 


^1 

pooled 


^11 

pooled 


^21 

pooled 


^31 

pooled 


• • • 


• • • 


• • • 


... 


A 


A 


A 


A 


^5 

pooled 


^ 15 

pooled 


25 

pooled 


^35 

pooled 


A 


A 


A 


A 


pooled 


Spooled 


Spooled 


M 1 o r 

pooled 


• • • 


... 


« « « 


• • • 


A 


A 


A 


A 


^ 10 pooled 20 pooled 


^30 

pooled 


Spooled 



Source Of Variation 



Degrees Of Freedom 



Programs 

Districts 

Programs x Districts 

Classes Within Programs x Districts 
Total 



3 

1 

3 

32 

39 
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Unique Applications of Multiple Matrix Sampling 



matrix sampling has been used traditionally to estimate 
parameters of standardized tests where the total test score is equal to 
the sum of the item scores. For investigations focused primarily on 
group assessment, multiple matrix sampling has been demonstrated empir- 
ically to be an important and valuable procedure. Multiple matrix sam- 
pling however, is applicable to a broader range of research problems 
than that suggested by the current literature. Four unique and important 
applications of multiple matrix sampling are described in this chapter. 

s is the case with most psychometric procedures and is certainly the 
case with multiple matrix sampling, the range of applications is deter- 
mined solely by the degree of inventiveness in the individual researcher. 



Design of Experiments 



In the evaluation of instructional programs, the pre-post paradigm 
s used frequently and, as is traditionally the case, an individual test 
is administered to all examinees at both the start and end c 
Given ap item population related to the instructional progr, 



instruction . 
under eval- 
the addi- 
tes t pre 
i with para- 
ding* A 
intermediate 



uation, , a research design such as this Is improved easily w 
tion of multiple matrix sampling. In place of using the sa 
and post, random or s tratif ied-random parallel tests are us 
meters for both tests estimated through multiple matrix sam 
procedure such as this could be expanded further to include 
testing using additional parallel tests. An example of a design such as 
this and one demonstrating the concomitant benefits is given by Osbum 
a a .. oama er < 1968 > • the evaluation of instructional programs it 

? ^ hat 3 researcher is seldom interested in individual test 

items, individual tests, or individual examinees but is interested pri- 
marily in group behavior over time with regard to some specified item 

s “ ch » multiple matrix sampling in conjunction with random 
or s tratif ied-random parallel tests is an ideal measurement procedure. 



Estimation of Covariance and Correlation Matrices 



Item and test covariance matrices (and, hence, correlation matrices) 
are estimated readily through multiple matrix sampling. A modified 
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sampling plan is required such that all possible pairs of items or tests 
are included in one or more sub tests or subbatteries. For example, con- 
sider estimating the elements in a covariance matrix for a 5-item test. 

To compute the covariance of Item 1 with Item 4, there must be a sub- 
group of examinees responding to both Item 1 and Item 4. If the examinee 
subgroup is sampled randomly from the population of examinees* C0V(1, 4) 
computed over those examinees is an estimate of C0V(1, 4) which would 
have been obtained by testing all examinees over both items. All remain- 
ing entries in the covariance matrix are estimated identically. A test 
covariance matrix is determined similarly with items being replaced by 
tests. A procedure such as this sets the stage for multiple matrix 
sampling playing an important role in a variety of multivariate procedures 
as, for example, factor analysis. Although little has been done in this 
area, some important preliminary research and a few of the relevant equa- 
tions for estimating parameters have been reported by Lord (1960), Ray, 
Hundleby and Goldstein (1962)* Knapp (1968) and Timm (1970) . 



Questionnaires and Surveys 

A perennial problem with ques tionnnaires and surveys is the disap- 
pointingly low rate of completions or returns. Return rates of 20 to 30 
per cent are not uncommon. Although examinees fail to return question- 
naires for a multitude of reasons, one factor is undoubtedly the length 
of the questionnaire and the time required to complete all questions. 

If the measurement required is the proportion of examinees in each cate- 
gory, results can be approximated through multiple matrix sampling by 
administering questions selected randomly to a random sample of examinees. 
For example, if an 8— page questionnaire were to be administered to all 
elementary school teachers within a particular city, the questions con- 
tained therein could be divided into 8 sub ques tionnai res (each of which 
would require no more than the front of one piece of paper) with each 
subquestionnaire administered to a random sample of teachers. The time 
for completing each sub ques tionnai re is minimal and, as such, may increase 
the rate of returns. The point to be made is simply this: a little data 

from a large number of teachers is better than a lot of data from few 
teachers. It must be remembered, however, that questions within question- 
naires are interrelated frequently (If "No” on Question 13, go to Question 
20.) and complications such as these must be incorporated in constructing 
sub ques tionnai res • 



Measurement in the Affective Domain 

It is frequently the case that an Investigator is interested in 
scaling the preferences or affect of a group of individuals for a par- 
ticular set of objects. Although there are several procedures which could 
be used, the method of paired-comparisons is one encountered frequently 
in the literature (e.g., Snider (1960) and Holliman (1970) . In the 
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method of paired— comparisons , all possible combinations of the objects 
taking two at a time are presented individually and for each pair the 
examinee is asked to indicate his preference. For example, if 6 stimuli 
were being scaled by the method of paired— comparisons , the test so con 
structed would contain 6(6— 1)/2 — 15 items > for 12 stimuli, 66 items. 

After all pairs have been administered to all examinees, the preliminary 
analysis of the data involves the computation of the F-matrix and subse- 
quent P-matrix. The P-matrix is the base from which the scale values per 
stimulus are computed and it is in estimating the values in the P-matrix 
that an application of multiple matrix sampling is found. Relevant pre- 
liminary research in this area has been reported by McCormick and Roberts 
(1952) , McCormick and Bachus (1952) and Bursack and Cook (1970) • If the 
s stimuli are numbered consecutively from 1 to s, the F-matrix is an 
s by s matrix with entries denoting the frequency with which the col- 
umn stimulus was judged more favorable than the row stimulus. An example 
of an F-matrix and associated P-matrix are given in Table 7.1. Dividing 
each entry in the F-matrix by the total number of examinees, which is in 
this case equal to 17, produces the corresponding entry in the P-matrix 
labeled appropriately as the proportion of examinees selecting the col- 
umn stimulus over the row stimulus. In estimating the entries in the 
P-matrix through multiple matrix sampling, paired-comparisons are selec- 
ted at random from the pool of all possible pairs and administered to 
samples of examinees selected randomly from the testable population. 

Shoemaker (1971) using a post mortem item-examinee sampling design 
has explored systematically the feasibility of using multiple matrix 
sampling— to estimate scale values obtained by the method of paired-com- 
parisons. The major conclusions reached in this investigation were that 
(a) scale values can be approximated satisfactorily through multiple 
matrix sampling, and (b) the similarity between the estimated scale values 
and the normative scale values increases with increases in the number of 
observations acquired by the sampling plan, with the converse true. The 
specific procedure used to estimate the P— matrix from subtest results is 
detailed in the following 5 steps. Each step is illustrated with results 
from one replication of a (3/10/15) sampling plan. (In the Shoemaker 
investigation, the data base consisted of responses made by 407 primary 
grade students to a 15— item test designed to scale degree of affect to 
6 stimuli.) 

Step 1: Three sub tests containing 10 items each are formed by sam- 

pling items randomly and without replacement within subtests but with 
replacement between sub tests. 




Subtest Items 



1 


8 


14 


11 


9 


6 


2 


3 


12 


15 


10 


2 


14 


6 


3 


12 


4 


1 


2 


11 


13 


10 


3 


14 


13 


2 


9 


6 


11 


4 


15 


3 


12 
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Step 2: Three subgroups of examinees containing 15 examinees each 

are formed by sampling randomly and without replacement from the 407- 
examinee population. 



Subgroup 

1 

2 

3 



Examinees 

359 22 280 272 139 206 169 321 323 23 271 66 221 109 100 
345 367 281 390 366 70 361 250 154 168 8 138 279 335 399 
342 220 276 125 382 219 217 327 401 385 113 62 77 192 156 



Step 3: Pairing subtest ji with subgroup jL, an f-matrix is formed 

for each subtest using only the responses made by the corresponding 
examinee subgroup on the items contained in that subtest. Each f-matr x 
is constructed in conjunction with a link-matrix containing the co e 
numbers of stimuli paired within each test item. For the data base 
considered herein, the link-matrix was 



Test Item Stimulus Pair 



01 


1 


2 


02 


4 


3 


03 


5 


6 


04 


2 


6 


05 


1 


3 


06 


4 


5 


07 


2 


3 


08 


1 


5 


09 


4 


6 


10 


1 


4 


11 


2 


5 


12 


3 


6 


13 


3 


5 


14 


1 


6 


15 


2 


4 



The f— matrices for the 3 subtests used in (3/10/15) are 





0 

0 

7 

0 

0 



>:> i- 



0 


9 


14 


8 


0 


8 


15 


0 




5 


0 


4 


10 




12 


9 


0 


3 




2 


11 


6 


13 





6u 



f-matrix 1 
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f -matrix 2 = 



f -matrix 3 = 





13 


0 


5 


0 


10 


2 




0 


0 


11 


4 


0 


0 




4 


6 


2 


10 


0 


11 




11 


0 


0 


4 


9 


4 




1 


5 


11 


13 


0 


14 





0 

0 

0 

0 

8 



0 

0 

6 

1 

7 



0 

0 

9 

9 

11 



0 

9 

6 

2 

4 



0 

14 

6 

13 

12 



7 

8 
4 

11 

3 



Step 4: In pooling the f-matrices to obtain the P-matrix, an ac- 

counting-matrix is required to distinguish between items omitted in the 
construction of sub tests and items to which all examinees in a pax'ticular 
subgroup responded identically. For the f— matrices given in step 3, the 
accounting-matrix is 



accounting-matrix - 



0 

1 

0 

2 

1 

3 



1 

0 

0 

2 

3 

2 



0 

0 

0 

3 

2 

3 



2 

2 

3 

0 

3 

2 



1 

3 

2 

3 

0 

3 



3 

2 

3 

2 

3 

0 



Off— diagonal zeros are of critical importance in pooling sub test results. 

In each f-matrix, f(i,j) + f(j,i) = n for those stimulus pairs contained 
within the subtest and f(j,i), for example, could be zero for two reasons: 
(a) the item containing stimulus pair (i,j) was not included in the sub- 
test, or (b) all examinees in that particular subgroup selected stimulus 
i over stimulus j. This distinction must be maintained in pooling the 
f-matrices to produce the P-matrix. 

Step 5: The P-matrix is formed by pooling across corresponding entries 

in tie f-matrices after each entry in the f-matrix has been divided by the 
number of examinees in the corresponding subgroup. The sum of proportions 
i<7 then divided by the corresponding number in the accounting— matrix. As 
an example, consider computing the (1,6) and (5,1) entries in the P— matrix: 



O 

ERIC 
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8/15 + 10/15 + 7/15 



.556 



P (1 > 6 ) 

3 



1/15 + no data + no data 



P(5,l) = — 

1 



.067 



If the n umb er of examinees per subgroup is unequal, the proportions 
are co mb ined by a weighted arithmetic mean and the corresponding entry in 
the accounting— mat rix is equal to the number of examinees for which data 
existed. Elements in the P-matrix are set equal to .5 if the correspond- 
ing entry in the accounting-matrix is equal to zero. In this example, 
the P-matrix is 



P-matrix 



500 


.867 


.500 


133 


.500 


.500 


500 


.500 


.500 


533 


.433 


.667 


067 


.111 


.600 


444 


.600 


.778 



.467 


.933 


.556 


.567 


.889 


.400 


.333 


.400 


.222 


.500 


.800 


.667 


.200 


.500 


.133 


.333 


.867 


.500 



After the P-matrix has been formed, scale values per stimulus are 
computed as if all examinees had responded to all items using computa- 
tional procedures detailed in Edwards (1957) . Using Thurs tone's Model V 
scaling procedure, the resultant scale values from the P— matrix given in 
step 5 and those obtained from vising all 407 examinees over all items 
are 





S 1 


S 2 


S 3 


S 4 


S 5 


S 6 


(3/10/15) 


.000 


.442 


.688 


.172 


1.184 


.184 


Norm (5) 


.000 


.075 


.638 


.215 


1.023 


.193 
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Parameters Through Multiple Matrix Sampling And For Approximating Norma- 
tive Distributions With The Negative Hypergeometric Distribution 
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A Fortran IV Program For Estimating Test Parameters Through Multiple Matrix 
Sampling And For Approximating A Normative Distribution Of Test Scores With 
The Negative Hyper geometric Distribution 



The negative hypergeometric distribution provides a reasonably good 
fit for a variety of test score distributions when the test score is the 
number of correct answers. The negative hypergeometric distribution is 
a function of the mean test score, the variance of the test scores and 
the total number of items in the test. The first two parameters may be 
approximated efficiently by multiple matrix sampling. Furthermore, the 
negative hypergeometric distribution with parameters estimated by multiple 
matrix sampling can be used satisfactorily to approximate a normative 
distribution of number correct test scores. 

In multiple matrix sampling, a set of K test items is randomly div- 
ided into subsets of items. Each subset of items is then randomly assign- 
ed to a group of examinees. Although each examinee receives only a pro- 
portion of the complete set of test items, the statistical model permits 
one to estimate the mean and variance of the total test score distribution 
for all examinees over the complete set of test items. Multiple matrix 
sampling is an efficient procedure for appro mating a normative distri- 
bution when it is not possible or is economically unfeasible to administer 
the complete set of K items to all examinees in the testable population. 

The Fortran IV program which approximates the normative distribution 
with the negative hypergeometric distribution is relatively machine- 
independent and has been implemented easily on an IBM 7040, IBM S360/50, 
IBM S360/91 and a UNIVAC 1108. The program has been designed to approx- 
imate test score distributions involving at maximum 500 items. However, 
this restriction may be easily modified. The number of subtests and 
n timber of examinees per subtest are limited only by the amount of computer 
time available. 



Organization Of Control Cards And Data Cards 



columns (all integers right-jus tified) 



Card 


Set 1 


(1 


card) 


1-72 


Card 


Set 2 


(1 


card) 


1-5 



6-10 

11-15 



Alphanumeric title of project 

Integer number of examinee groups 

Integer number of items in each 
sub test 

Integer number of examinees per 
subgroup 
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columns 



(all Integers right- justified) 



5 I -55 Punch 00000 if there is only one for- 

mat card by which all 
• item scores are to be 

inputted 

Punch 00001 if there is to be a dif- 
ferent format card for 
each item-examinee sam- 
ple within a data set 

Standard Fortran IV format punched 
in columns 1-72 on each card and ; 

enclosed in parentheses for inputting; 
item scores for each examinee in each 
item- examinee sample. The number of 
format cards may not exceed 9 for 
each item-examinee data set. The 
first card after the format cards 
must contain END OF FORMAT in col- 
umns 1-13. 

Example: (5X, 25F1.0) 

END OF FORMAT 

Data Card Set The responses of each examinee per 

item-exattiinee sample -l be se— 

(k cards , optional) quenced by examinee group and within 

each group by examinee. 

Acceptable Input Data Structures 



Format Card Set 
(k cards, optional) 



PI am 1 



Plan 2 



Plan 3 



Fortran Source Deck 
Card Set: 1 
Card Set 2 



Fortran Source Deck 
Card Set 1 
Card Set 2 
Format Card Set 
Data Cards 



Fortran Source Qedfc 
Card Set 1 
Card Set 2 
Format Card Set TL 
Data From Sample 1 
Format Card Set 2. 
Data From SantpAe 2 
• • • 

Format Card Set tat 
Data From SampAe t 
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Plan 4 



Plan 5 



Fortran Source Deck 
Card Set 1 
Card Set 2 
Format Card Set 
Card with no. of examinees 
and items for sub test 1 
Data from sub test 1 
Card with no. of examinees 
and items for subtest 2 
Data from sub test 2 



Card with no. of examinees 
and items for sub test t 
Data from sub test t 



Fortran Source Deck 
Card Set 1 
Card Set 2 
Format Card Set 1 
Card with no. of examinees 
and items for sub test 1 
Data from sub test 1 
Format Card Set 2 
Card with no. of examinees 
and items for sub test 2 
Data from sub test 2 



Format Card Set t 
Card with no. of examinees 
and items for sub test t 
Data from sub test t 



Plan 1: 



Mean and variance of test scores are 
No item scores are required. 



inputted on card set 2. 



Plan 2: Mean and variance of test scores are to be estimated from item- 

examinee samples. All item scores in each item— examinee sample 
are organized in the same manner on the data card and are to be 
inputted with one format card. 

Plan 3: Same as Plan 2 with exception that item scores for each item- 

examinee sample are not organized on data cards in same manner. 

Each sample requires an individual set of format cards describing 
how item scores are organized for that particular sample. 

Plan 4: Same as Plan 2 with exception that number of examinees and number 

of items per subtest are not constant across subtests. Same format 
card is used for each data set. 



Plan 5: Same as Plan 3 with exception that number of examinees and number 

of items per subtest are not constant across sub tests. In addition, 
different format cards are used for each data set. 





a n o or, c: 



-71- 






■ « ■«• «• U -B- <>!>■•»«■ «■ G 



;; a PPRO x 1 MA T ION OF F H £ 0 UE NC y D1STRIBUII0N OF TFST SCORES 
i 0 y NEGATIVE HYPE ROE OMbTR I C DISTRIBUTION 



C reference 



c 

r • 



lord, f,m. AND NOVICK, 
SCORES. READING, 



M.K, STATISTICAL THEORIES OF MENTAL. IE S T 
MASS, AUP ISON -WhSLtY* 19 6B , CHAPTER 2,5. 



C 

0 DAVID M. SHOEMAKER 

c 

j NTS = NUMBER OF ITfcMS PER SUBTEST 

C NTP = NUMBER of ITEMS IN TEST ITEM POPULATION 

C NS M = NUMBER OF SUBTESTS 

o nss = number of examinees per subgroup 

r. NS P = NUMBER OF EXAMINEES IN EXAMINEE POPULATION 

c /Bar = ESTIMATE of mean test score 

c VAR = ES T I MA rt OF TEST SCORE VARIANCE 



a-' 0 ' ** ** ** » ** 






COMMON DUMMY ( 500 ), P ( 50 0 ) 

DIMENSION TITLE (18) 

G 

>:"! INPUT PROBLEM PARAMETERS 

C 

1000 READ < 5, 1* EN|j = 5000 ) ( T I T L E ( I > , I = 1 » 1 8 ) » 

1NSM , NT S, NSS , NTP f NSP , XBAR ; VAR ,NGPH NF MT 
WRITE (6.5) (TlTLE(t), 1 = 1, 18) 

IF ( l F IX ( V A R *1 0 0 0 . ) ■ . NE . 0 ) WRITE (6,9) XBARtVAR 



ESTIMATE MEAN AND VARIANCE FROM SUBTESTS 

IF ( I F I X ( V A K * 1 U 0 0 » ) • E Q , 0 ) CALL POOLtNSS.NTS.NSM.NTP.XBAR.VAK.NF 
IF ( NSP ..EO. 0 ) NSP = 10Ua- 
WP I rt (6,10) NSP 

COMPUTE PARAMETERS FOR NEGATIVE H YP ER GE OM ET R I C DISTRIBUTION 



S = N T P 

A 2 1 = (S/(S-1. ) )*(1.- XBAR *’(S- XBAR)/ (S*VAK)) 

IF ( A 21 . GT . 0. ) GO TO 40 

WRITE (6,7) API 
GO TO 1000 
40 continue 

A= <-l, +1 ./A21) *XBAR 
B=“A w l.+S/A21 
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SL 0G1 = 0 , 

SL OG 2 = 0 • 

C = A +B 

DO t>Q 1 = 1* NT P 

SlOGl=SLOGl + ALOGl0 (y * l4 'l* ) 

■PQ SLOG 2= SLOG 2 * Al OGlO <C»I +1 , ) 
C510.«h»(GLUG1-GlOg2) 

WRITE (6*3) A21.A.B»C 

c compute NEGATIVE HYPERUEOMfc TR iC DISTRIBUTION 



N3sNTP+1 
WRITE ( 6 * ) 

CK=0 , 

DO 10Q I=1*N3 
K = 1 ~ 1 

CALL NEGHGH (K*A»B»C*S»NSR*HX|HFX) 
CK sOK+HX 
P( 1 )=HX 

WRITE (6*2) K.HX.HFX.CK 
100 CONTINUE 



C PLOT NEGATIVE HYPbRGEOMETR I C DISTRIBUTION 

1 IF ( NGPH .£Q, 0 ) CALL PL.OT (NTH) 

GO TO 1000 
SOOO WRITE (6*6) 
call EXIT 

F ORMAT (18A4/5I5. 2H0 . 0*215) 

f-URMAT ( 1 1 0 . 3F 30 . 7 ) , K „ r4fi ,.„ v XNf , P1M 

FORMAT ( / 7H KR21 sF 12 < 3 »6X» 3HA =F 15 .7/25X ,3HB = F15. //25X, 3HC =E18 

1 FORMAT (////5X*5HSC0RE»22X,4HH(X) , 26X , 6HN#H < X ) , 24 X . 6HCUM MX//) 

FORMAT M3H , ^5i / NEGATIVE OR ZERO ... BATA SET ABORTED . 5X . 6HKR21 

P 0 R M A T (1H-»2o x ,19HaLl INPUT pROCES§bD* 

FORMAT ( // 7H XBAR ' Si- 12 ,3//7H VAR =Fl2.3) 

FORMAT ( / 4 X i 3 H N =18) 

END 



1 

2 

3 

4 

5 

7 

8 

9 

10 
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SUBKQUTlNfc. NEGHGR ( K , A , B , C , S , NSUB » HX , HF X ) 
C 

C NEGATIVE HYPERGEOMETRIC FUNCTION 
C 

j F ( K .EG. 0 > GO 10 1*0 

SL UG 1 = 0 , 

51.00 2 = 0 . 

Sl 003 = 0 . 

SL 004=0. 

DC 100 1=1 »K 

SLOG 1= SLOG i + At OGlO ( S - 1 +1 . ) 

Sl. OG2 = SLUG2*ALOlilO<A + I- l . ) 

SL003 = SL003-*-AL0010(0"l' f l. > 

1 0 () bLOG4 = SLOO4 + ALOO10 (KLOAT ( l ) ) 

HX=C*10 .*» (SL0G1+SL0G2-SL0G3-SL0G4 ) 

1 25 HFX = HX*NSUB 
RETURN 
150 HX=C 

GC TO 12* 

END 



SUBROUTINE RDFMr (FMT ) 

c SUBROUTINE F OR INPUTTING VAR I ABLE FORMAT 

C 

c INPUT STRUCTURE 

C FORMAT. (ENCLOSED IN PARENTHESES) COL 1-72 

C CONTINUE ON CARD 2 IF NECESSARY 

C CONTINUE ON CARD 3 IF NECESSARY 

C ETC. 

c maximum number of format caros is 9 

c 'END OF FORMAT' NECESSARY ... PUNCH IN COLUMNS 1 

C 

DIMENSION F M T ( 20 o ) 

DATA END/3HENI)/ 

N= 1 

DO 100 1=1*10 

M = N ♦ 1 7 

READ (5, 1) <F M I ( J> , J = N,M ) 

IF ( FMT(N) .EU. END ) RETURN 
100 N = N ♦ 1 8 

WRITE (6,2) 

STOP 

1 FORMAT (18A4) 

2 FORMAT (3/H EXCESSIVE NUMBER OF FORMAT CARDS) 

END 
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SUBROUTINE POOL ( NSUB . N I T£MS , NS AM , NTP , XfcjAR , VAR , NF MT ) 

:Tl RM 1 NAT I ON OF POOLED ESTIMATE OF POPULATION MEAN TEST SCORE AND 



COMMON P ( 500 ) » X ( 50 0 > 

dimension F M T ( 2 0 0 ) 

IF ( NFMT .EQ. U ) CALL, RDFMT(FMT) 

WRITE ( 6 . 1 ) '■! 

NT EST=NSU8#NI TEMS / 

StSTM = 0 . 

S F S T V = 0 . 

NSM = 0 

SWGH |=o. 

i:o iooo 1 = 1. ns am 

if ( NFMT .NE. 0 ) call rufmkfmd 

Ip ( MTjfST . EQ. 0 ) P£Au (5,6) NSUB,Ni1e m S 



5 Y T = 0 , 

i.i.i 5 o J = i* Nl rt MS 
P ( j ) = 0 . 

r.l. 5 0 0 J = l,NsUU 

Ri.AU ( 5 , F M T ) (X (K ) , K=1,NITEMS ) 

Y = 0 . 

DU 510 K=1.N ITEMS 
i (K)=P(K)+X(K) 

Y = Y ♦ X ( K ) 

•j Y = S Y + Y 

i SY y =SYY+Y*Y 
XBR=SY/NSJW 

VR=(SYY-SY*SY/NSUB)/NSUB 

5PQ=0. 

i; 0 520 J s l»N]TEMS 
PP =P ( j )/NSUd 
) SPQ=SPQ+PP*( 1 . -PP ) 

NS M = NS M+ NS Utl 

'W(JHT«NSUB*N items 
F-STm' = NTP*X8R>N I T EMS 

LS I V:=<NSUB*N'TP*-( (NTP-1 . )*VR-(NTP-N1TEMS)*SPQ) )/ 

1 (N ITEMS* (N I TEMS-1 . )* (NSUB-l. ) ) 

SESTM=SESTM+EST M*WGHT 
SES T V = SES T V + ES r V*WGH T 
SwGHT=SWGHT+WGH T 
WR l I E (6,2) I.ESTM.ESTV 

00 continue 

XB AH =SES TM /S WGHT 
VAR=SESTV/SWGH F 

IF ( NSM , L T . 500 ) VAR = VAR* ( NSM-1 . ) /NSM 

WRITE (6*3) XBAR.VAR 

FORMAT ( ////24 X , 21HEST l MATE OF P AR AMETER///5X , 6HSAMPLE » lOX , 4HMEAN 
1 , 16X , 8HVAR IANCfc/// ) 



! A N C E 




'F 20 . 7//18H POOLED VAR I ANCE 




F NO 




ERIC 
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SUdHOUTIMt. PLOT (NlTfcMS) 

REAL N 

COMMON N ( 5 0 0 ) ,P<500) 

D i MbNS I ON HOD(IO) 

DATA BCU/1HI) , 1H1 , 1H2, lHvi , 1H4 , lHb , 1H6, 1H7 , lHtt, 1H9/ 

DATA 9LK , POT » XX/1H .1H..1HX/ 

C 

C LOCATE MAXIMUM VALUE FOR H ( X ) 

c 

NMNsN [ TEMS+ i 
T=U. 

DO 5 0 1=1 * N N N 

IF ( P(I) .OT . T ) TaP( I ) 

50 CONTINUE 

c determination of appropriate scale factor for h<x> plot 

c 

J = 0 

DO 60 1=1.6 

K = I -1 

J=T*10,**K 

IF ( J ,bO. 0 ) GO TO 60 
J=K-1 * 

WRITE (6.3) J 
GO TO 70 
60 CONTINUE 
C 

U SCALE H ( X ) BEFORE PLOTTING 
C 

/0 DO 75 1=1. NNN 
75 P ( l ) =P ( I ) *10 . **J 

C ' . ... 

C LABEL ORDINATE 
C 

WRITE (6.1) 

DO 500 1=1.100 
500 N(I)=BLK 

N ( 101 ) =BC J ( 2 ) 

WRITE (6.2) (N( J) . J=l. 101) 

NMO 

DO 550 1=1.10 
1,0 5/5 J=l»10 
575 N(NN+J)=BCU( I ) 

NN=NN+10 
550 CONTINUE 

N(101)=8CU(1) 

WRITE, (6.2) ( N( J) » J=1 ,101 ) 

DO 500 1 =1.101,10 ’* " 

DO 590 U =1.10 
K = J-l 





‘WOT*** 
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9 5 



N (101 ) = 8t;U< 1 ) 

wk 1 r e (6,2) ( N ( J ) » J = 1 » 3.0 1 ) 
DO 595 1=1.101 
N < 1 ) = 1)01 

k r« l r t ( 6 » 2 J (N(J) 



r; ploi values ot scaled h<x) 

c 

DO 1 0 D 1 = 1# NNN 
* = 1 " 1 

L = P ( I )*1U0 .+1.5 
DO 105 J = 1 # 1 0 1 
105 N (J)=BLK 

DO 1 10 J =1 #L 
ll 0 ( J ) = XX 

DO 120 J=ll»l01»l0 

if ( N ( J ) .bO. BLK ) N ( J ) = UO T 
120 CONTINUE 

WRITE (6*4) K,(N(J) ,J-1»1Q1) 
10 0 CONI 1 NUE 
C 

0 END OT GRAPH 



900 



1 

2 

4 

4 



DO 9 oO l=l.tol 
N ( 1 ) =DOT 

WRITE (6.2) (N( J ) . J=l# 101) 

FORMAT ( 1H1. 5QX. 40HPROPORT ION OF POPULATION RECEIVING SCORE//) 

FORMAT ( ////5X^19HH( X) SCALED BY 10 EI3.9H IN GRAPH//) 

FORMAT ( I 10 ,4X# lOlAl) 

END 




j . 





wmgsma usaai 
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Sample Output 



Card 

column 



00000000011111111112 22 2222 22 23 33 33 33 33 34 4444 44 4 4 45 5555 55 55 56 

12345678901 23 45678901 23 4567 89 Ql 23 4567890123456/89 01 23 4567890 



FIRST Y£AR WORD SHELLING PROJfcCl 

00005000000000000050 

(lOX.lOFl.O) 



END OF ToRhAT 

oooienooio 



01 


1 1 


lOlllllHO 


0 2 


1 1 


HHlOl OH 


03 


1 1 


1110100000 


04 


1 1 


1110000000 


05 


1 1 


0000000000 


06 


1 1 


llOtOlOOOl 


07 


2 1 


111010101 0 


0 0 


2 1 


1100100000 


09 


2 1 


0 0 0 0 o 1 1 0 1 0 


1 0 


2 1 


0100000000 


11 


2 1 


11 11 1.0 10 10 


12 


2 1 


1001001000 


13 


3 1 


010010 onoo 


14 


3 1 


01.11100010 


15 


3 1 


0000000000 


16 


3 1 


oooooooooo 


17 


3 1 


1101001110 


18 2 1 
0001400010 


oooooooooo 

1 


01 


1 2 


1111110011 


0 2 


1 2 


10H100000 


03 


1 2 


11 0010 10 uo 


04 


1 2 


1111110000 


05 


1 2 


101111001.0 


06 


1 2 


H1011000 0 


07 


2 2 


1Q101Q0000 


08 


2 2 


111111001 0 


09 


2 2 


11011110 10 


10 


2 2 


1111110000 


11 


3 2 


oooinoooo 


12 


3 2 


1111111100 


13 


3 2 


1101Q00000 


14 


3 2 


OQlillOOOO 



SHOE MAKER/ OK AU A 

00000 





■'Si* 






f 
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Card 

column 



000000000111111111122222222223333333333444444444455555555556 
1 23456 7890123456 7890123456 7890 123456789 01234567890123456 7890 



0001300010 




01 . 


1 3 


1101111101 


0 2 


1 3 


OlOlOOlOoO 


03 


1 3 


1001011101 


04 


1 3 


0010010000 


05 


1 3 


UllUllU 


O 6 


1 3 


llfjlllolpB 


0 / 


2 3 


0100000100 


OB 


2 3 


01111.11110 


0 9 


2 3 


lOOOOuOOOO 


10 


2 3 


1001000000 


H 


3 3 


liunnoo 


12 


3 3 


nil 000100 


1 3 


3 3 


0000000000 


0001300010 


01 


1 4 


llOlOUllOO 


0* 


1 4 


QOlOuOlOoO 


0 3 


1 4 


llHull 01 1 


04 


1 4 


1111111111 


05 


1 4 


mi oimi 


06 


2 4 


OUIOIOOOO 


07 


2 4 


li U om n 


08 


2 4 


0 0 0 0 0 0 0 0 0 o 


09 


2 4 


Olio U 01 100 


1 0 


3 4 


0011001000 


11 


3 4 


uuoouio 


l2 


3 4 


00 3 1 00 1 0 00 


13 


3 4 


0000000000 


0001 200010 


01 


1 5 


1001001010 


02 


1 5 


oooooooooo 


03 


1 5 


loo oooooon 


04 


1 5 


oooooooooo 


05 


2 5 


0101111010 


06 


2 5 


1110010100 


07 


2 5 


oooooooooo 


08 


2 5 


oooooooooo 


09 


3 5 


1011111100 


10 


3 5 


oooooooooo 


11 


3 5 


oooooooooo 


12 


3 5 


oooooooooo 






mm^im 
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APPENDIX B 

Computer Program For Simulating Multiple Matrix Sampling 
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Computer Program For Simulating Multiple Matrix Sampling 



computer program for simulating multiple matrix sampling is de— 
scribed in detail in Chapter V. A listing of the Fortran IV program is 
given in this appendix for those readers who may want to implement the 
model on the computer configuration available to them. The program given 
herein was written originally for a UNIVAC 1108 and a modified version 
has been implemented on an IBM S360/91. In modifying the program for the 
S360/91, the only changes made were those involving the uniform (.00 to 
.99) random number generator RUNIF. On the 1108, RUNIF is initialized by 
RINITL. Calling RINITL with BASE as the argument causes BASE to be used 
as the starting value or seed in the algorithm used by RUNIF in generating 
uniform random numbers. Because RINITL is specific to UNIVAC 1108, readers 
should consult the local computing center staff to determine the subpro- 
gram and calling procedures at that installation comparable to the RINITL/ 
RUNIF system. The conversion process was relatively simple for the S360/ 
91 and it is anticipated that such will be the case with other hardware 
and software systems. Input values to the program are made on one para- 
meter card. The organization of the card is described at the beginning 
of the program listing. Examples of parameter cards are found on page 
97 of this appendix. Sample output from the program is given in Chapter 

V. 
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c o o n o o c) o 



' ' tWffWf 1W4' ■ 












x»sisi(i*w?w'-'-*« «■ «•: ■ f,Wiyt>^.) m»nw L - " " < *^ j| - Mfr****** 



"(WWWWM^’ 
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c 



^tt********#****^*****************’********* # ****** # *************'*' ,, ^***** 



C 

C 

C 

C 

c 

c 

c 

c 

L 

c 

c 

c 

c 

c 

c 

c 

c 

0 

c 



COMPUTER SIMULATION OF ITEM-EXAMINEE SAMPLING 
DAVID M, SHOEMAKER 



PARAMETER CARD (THERE IS JUST ONE) 



COLUMNS 

01-03 



(ALL INTEGERS R I G HT - J US T I F I ED ) 

INTEGER NUMBER OF ITEMS IN TOTAL TEST 



0 A- 09 



10-15 



DESIRED MEAN TEST SCORE IN POPULATION 
(MUST BE SPECIFIED. WITH DECIMAL POINT 



PUNCHED ON CARD) 



DESIRED VARIANCE OF TEST SCORES IN POPULATION 
(WITH DECIMAL POINT PUNCHED ON CARD) 

NOTE ... IF VARIANCE IS OMITTED. RELIABILITY MUST 
BE SPECIFIED. 



c 

c 

w 

c 

c 

c 



16-21 



DESIRED VARIANCE OF ITEM DIFFICULTY INDICES OVER 
POPULATION OF EXAMINEES, THE I TE M D I FF I CULTY 
INDEX FOR ITEM I IS THE PROPORTION OF EXAMINEES 
ANSWERING ITEM I CORRECTLY. 

(MUST BE SPECIFIED WITH DECIMAL POINT PUNCHED ON CARD) 



WITH SKEWED DISTRIBUTIONS. VARIANCE OF ITEM 
DIFFICULTY INDICES IS ASSUMED TO BE EQUAL TO ZERO. 



22-27 



C 

c 

c 

I., 

c 

c 

c 

c 

c 

c 

C 

c 

c 

c 

c 



o t 

ERICJ 



2B-31 

32-35 



36-39 



40-43 



44 



DESIRED RELIABILITY OF TEST SCORES IN POPULATION 
(WITH DECIMAL POINT PUNCHED ON CARD) 

NOTE ... IF RELIABILITY IS OMITTED, VARIANCE MUST 
BE SPECIFIED. 

INTEGER NUMBER OF SUBTESTS IN ITEM-EXAMINEE SAMPLING 

INTEGER NUMBER OF ITEMS PER SUBTEST 
(CONSTANT ACROSS SUBTESTS) 

INTEGER NUMBER OF EXAMINEES PER SUBTEST 
(CONSTANT'ACROSS SUBTESTS) 

INIEGEK NUMBER OF INDEPENDENT REPLICATIONS OF ITEM- 
EXAMINEE SAMPLING PLAN' 

SAMPLING PLAN FOR ITEMS 

0 * SAMPLING WITH REPLACEMENT 

(USED WHEN TK IS GREATER THAN K) 

1 s SAMPLING WilHOUl REPLACEMEN I 

* ' ( USED WHEN ,TK IS LESS THAN OR EQUAL TO K) 

se' 
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2 = SAMPLING WITHOUT REPLACEMENT BUT SUBJECT 

TO RESTRICTION THAT ITEMS OCCUR WITH EQUAL 

FREQUENCY AMONG SUBTESTS 

(USED WHEN TK IS GREATER THAN K) 



0 = NO INTERMEDIATE PRINTOUT 

1 s INTERMEDIATE PRINTOUT WANTED 

NEGATIVE HYPERGEOMETRIC DISTRIBUTION OPTION 

0 = NO NEG. HYPER. DIST. WANTED 

1 s COMPUTE NEG. HYPER. DIST. 

DEGREE OF SKEWNESS IN NORMATIVE DISTRIBUTION 

1 s NORMALLY DISTRIBUTED 

2 * POSITIVELY SKEWED 

3 = NEGATIVELY SKEWED 

SEED FOR UNIFORM RANDOM NUMBER GENERATOR (ODD NUMBER) 



45 



INTERMEDIATE PRINTOUT OPTION 



GENERATE ITEM DIFFICULTY INDICES 



0 = GENERATE NEW ITEM DIFFICULTY INDICES 

1 = USE ITEM DIFFICULTY INDICES GENERATED BY 



PREVIOUS DATA CARD 



RfcSTR ICTIONS 



MAXIMUM NUMBER OF ITEMS IS 150 (EASILY MODI T l ED # HOWEVER) 
ITEMS SCORED D l CHOTOMQUSLY 



PROGRAM WILL PROCESS REPEATED PARAMETER CARDS (NUMBER LIMITED ONLY 
BY AMOUNT 0! COMPUTER TIME ALLOCATED) 




real n.m.mpop 

COMMON N ( 150 ) . M ( ISO ) , L T ( 30 OU ) 

COMMON / d L 0 C K 1 / YB AR , YSD , MPOP , SPOP . K POP . ND I ST , BASE . I NT PRT 
COMMON /BLOCK?/ RND(150) 

COMMON /BLOCK 3/ P ( 1 50 ) # 0 ( 15 0 ) , NSUB 



2000 r ' ^ \ i/uno m un d i/D'iD DA/ A w. A?n. MI . T PT . Ivl SP T • (\1K S r 




INITIALIZE RANDOM number GENERATOR (UNIQUE TO UCC) 



ALL R 1 N I TL ( BASE ) 
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LJIj L> CJ 



wwwfTfWSW 
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C 

C 

MSUB=0 

C 

C .CHECK ON PARAMETERS 

c 

If ( ISAMP .NE. 1 ) GO TO 30 
IF ( N T* I P T . 61 . KPOP ) 60 TO 5H 
30 IF ( A 20 .LT. 0. .OR, A20 ,GT, 1. ) GO TO 55 

IF ( NDIS1 . G T . 3 .OR. NdIST .LT. 1 ) GO 10 55 
IF ( IFIX(A20*1000. ).EQ.0.AND.IFIX(VPOP*1000. J.tQ.O) GO TO ob 
[F ( IF I X (MPOPa 10 00 . ) .Ge . KPOP *100 0 ) GO TO 55 
IF ( PVAR .LT. .2 ) GO TO 70 
55 WR I T £ (6 ,3 ) 

Go TO 2000 
C 

c COMPUTE NECESSARY PARAMETERS 
C 

/0 T E MpsmPoP#C KPqP-MPOP) -KPOP*kPOP «P VA R 

If < I F I X ( VP OP ) . E U . 0 ) VPOPsTEMP/(KPOP-(KPOP-i,)*A20). 

SF0P = SQR1 ( VPOP ) 

WRUE (6,13) SPUR 

c ' 

IF < ISAVE .Eq. 1 ) Gq TO 102 



GENERATE ITEM OIFh ICULTY INDICES (PROPORTION OF EXAMINEES ANSWERING 
I TE-M CORREC (L Y ) 



IF < I F I X ( < KPOP-MPOP >»10Q0 . ) . EQ . 0 ) GO TO 55 
D 1 = 1 0 0 0 . 

pbar=mpop/kpop 

DO 173 1=1. KPOP 
173 0 ( I ) =0 . 

IF ( IF] X(PVAR*1000 . ) . GT . 0 ) GO TO 66 

DO 65 1=1. KPOP 



65 PClJsPBAR 
GO TO 102 

66 PSD=SQRT (PVAR ) 




C 





DO 100 I J = 1 » 100 
DO 74 1 = 1 . KPOP 
CALL RANUNU <Z> 
Q(I)-Z*PSD +PBAR 






IF ( Q ( I ) .LT. 0 . 


) o<n=o. 


74 


IF < 0 ( I ) .GT. 1 . 


) Q < I ) = 1 , 


C 




- ’ / 



L. > 

C DETERMINE INITIAL MEAN AND VARIANCE OF 
C INDICES 

C . \ 



SP = 0 . 

SPP=0. 

DO 81 1=1. KPOP 
PP=Q( I ) 

SP = SP«-PP 




GENERATED ITEM DIFFICULTY 



/ 
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81 SPP=SPP+PP»PP 

PVR= (SPP-SP*SP /KPOP ) /KPOP 
CVK=SQRT(PVAH/PVR) 

1 SCALE VARlANUt 01 [TPM DIEF ICULTY INDICES TO STANDARD 

C 

ap = 0 . 

SPP=0 . 

C O 8 2 1=1* KPOP 
Q ( 1 ) =Q ( I )*CVR 

lf(Q(I>.GT. 1. )U(i)=l. 

PP=Q ( I ) 

SP=SP+PP 

82 SPP=SPP+PP«PP 

PVR = < SPP-SP*SP /KPOP ) /KPOP 

■; SCALE MEAN OF ITEM DIFFICULTY INDICES 

D4=SP/KP0P-PH AH 
SP = U , 

SPP=0 , 

DO 84 I -1. KPOP 
Q ( I ) = Q ( I ) ”04 

IF ( 0(1) . LT . 0 . ) Q< I > =0 , 

IF ( Q( 1 ) • «T . 1 . ) Q< I >®1. 

PP = Q ( I ) 

SP-SP+PP 

84 SPP-SPP+PP*PP 

PVR= ( SPP-SP*SP/ KPOP ) /KPOP 

Pi:R=SP/KPOP 

D2=ABS< PVAR-P V* > 

03 =A BS (P8AK-PBK) 
il=(Dl+.0005)*1000. 

I 2 = ( D2V. 00 02 )* ID 00 . 

I 3= ( D3* . 0008 ) #1000 . 

IF ( 12 .Lt. D .AND, 13 ,LE, 5 ) GO TO 103 

IF ( 1 2 . (jE. 11 ) 00 TO 100 

D1=D2 

HO 90 1=1, KPOP 

VO P( I ) =Q ( 1 ) 

100 CONTINUE 

103 DO 1q4 I =1 ,KPOP 

104 P(i)=Q(I) 

NSrOp=KPOP-l 

DC 110 1=1 , NSTOP 

-w* = l ♦ 1 

*IF ( 1 IFI X( P( I ) *1 00 0. ) . Gfc , IF IX ( P < J )* 10 00 . ) ) GO TO 1 lO 

T t Mp = p( J) 

P ( J > =P ( I ) 

P( I ) = TEMP 

102 I F N ( 1 I NTPK I . FU . 1 ) WRITE (6*4) ( P< 1 ) » I -1 .KPOP) 

L 0 COMPUTATION OF CONSTANTS FOR GENERATION OF LOGNORMAL DIStRIBU I ION 

ERJC ; ^ ‘-'IP 



n o n c* n 









ir < nb ist .eq. i > go to m 

IF ( NDIST .EO. 3 ) MP OP =K PO P- MP OP 
YVAR=ALOG ( VPOP/ (MPOP*MPOP )♦! , ) 

Y8AR = AL0Ci( MPOP )-YVAR/2 . 

YSU = SQRT C YVAR ) 

C 0 M P U T fc ROUNDING VALUES FOR EACH TEST SCORE INTERVAL 
111 IF < ISAVE .EG). 0 ) CALL ROUND 

REPLICATION 01- ITEM-EXAMINEE SAMPLING PARADIGM 



DO / oO 0 I JK 5 1 1 NR EPS 
call alloc (nt, ipt, iSamp) 

IF ( INTPKT , E 0 . 0 ) GO TO 113 

WRl TE (6.5) 

J = 0 

Nl=NT*IpT 

DO 112 I =1 ,Nl, IPT 

KK-I+IPT-1 

J-J+l 

1 1 2 WRITE (6*6) J,(LT(K),K = I»KK) 

113 CALL POOL (NsP 1 , IPT. NT ,XBAR, XVAR > 

XS D= Q , 

IF ( XVaR .GT. 0. ) XSD=SOR T ( XVAR ) 

IF ( I NT PR T .EO. 1 ) WRITE ( 6*7) 1 JK « XBAR , XSD 

SXM=SXM*X8AH 

SXS=SXS+XSD 

SXXM=SXXM*XBAR»XBAR 

SXXS=SXXS+XSD»XSD 

C COMPUTATION OF CONSTANTS FOR NEGATIVE HYP6RGE0METRIC DISTRIBUTION OPTIO 



IF ( NHPER .EU. 0 ) GO 10 7000 

A21= (KPOP/ (KPOP-l . ) )*(1.-XBAR*< K POP- XBAR ) / ( K POP* XV AR ) ) 
I F ( A 21 . G I . 0 . ) GO TO 120 

WRITE (6,8) A 2 1 
GO 10 2000 

120 A=(-1.*1./A21)*XBAR 

Br-A-1. ♦KP0P/A21 
SL OG1 = 0 . 

S L. 0 G 2 = 0 . 

C = A ♦ B 

DO 140 1=1, KPOP . ' .. 

SLOG1=SLOG1+ALOG10 <8-1 ♦!> ) 

140 Sl. OG 2= SLOG 2 + A LOG 10 <C-I>1 . ) 
r = 1 fl . »» ( SLUG 1 -SLOG 2 ) 



SXM = 0 . 
SXS=0. 
S X XM=0 

sxxs=o 



c 
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>n 



N3=KP0H+i 
CK=0 . 

L 0 1 6 u 1 * 1 * N S 
K= l-l 

LALl N E GH G K ( K t A t B * C > H X ) 

c;k = ck-*-hx 

t.-Kl TL(6 ,lu) K.-HX.CK 



0 06 CON ! I NUE 

COHPlUE STAivuARD ERRORS OP ESTIMATE OVER REPLICATIONS 



RAWS =SXS/i\iRbPS 

HARM - 3 X M / N R.H P S . : - P," . H nr, 

3r S - SORT ( ( SX XS -SXS*SXS /NREPS ) /NR.tPS > 
h r M> s (jR T ( ( s XX H -SXM^SXM / NREP S > / NRt R.S ) 
(6*11) UAKM.SEM, BAR'S, SES 
GO 1 0 2000 



g X I 1 gr acefull y 



0 0 ti W-R I T E ( 6 * 1 2 ) 
call EX 11 



FORMAT ST ATE MEN I S 



;ri'4 //«: 4X. M//4X.<IHVil'T.3X.14//4X.5"SI<et'3.2X. 

1 U////QU swi t r H E S / / / 4 X * Id Ml TEM- SAMP LING PLAN * 8X , I 2//-4X , 23HNEUA JIVE 
^ HYPERGEOMETpfc. 3X , 1 2//4X, 22HN0RMATI Vfc U I S T R I HU T I ON , I 6 / / 4X , 2 1 e 

6 F0rS!^ T (2dH l *n U ERio>/oN PARAMETER C, A R D / / / 1 2 X » 2 HO H / 7 2 9 H NEEL) MURE 

"FLRMArc/J/^A^I^M DW.t^Y Vi N 0,1 CE S/ / (j;10 F t ... 

FORMAT' < / / / / 8 H SU B I.6ST , 5.X .SH^TEMSV / / ) ^ : - 

F O HM A T ( IX » 1 5 , 2 X , 2 0 I 4 / ) ) . 

f (.RMAT (//.16H R E p L, I C’A ;Ti tiV; N 5 ’» 5 X , 

1 E S 1 . OF SI ANDARD D EVL/LTrt O N;|P^ ! 0^ / / > 

FORMAT ( 4 sSH RR21 NEGfTFl^|b" 

If 10. 4) 















Si® 



F- nKMA T f PROHlFm' NO ! F 9. 1 '// )sOH PAR AMt T £ RS 1 N P U T.TE D// /4 X . 4 HMEAN . 

?* - »» ? ••? ?> sail 



< V- 



• ■ 1 • V.V j ?viiS 4 - 






* 5.X, SH, ITEMS///) - ^ 

1 QN X NO^! j 5.! 5X , 1 2HES T . OF MEAN * P 24 . 3/ 26X* 26H i 

Tl ON * F 10 .6 / / ) ; • ' / '/- J *• * , ' 

■H W it*. ... UA1A StT ABOHT fc-.U » 5X . 6HKR21 * ^ 

: : . - y > . . : : - ; yF : ;; ■■ y ; y : 'v Fy' .'.' y y ' ' :y 1 yy^ 
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i: 

C 

c 

c 



SUBROUTINE POOL (NSPT » IPT iNT i XBAR > XV AR ) 

determination ok pooled estimate ok population mean test scope and 

VARIANCE 



real mpop 

COMMON P( 150) ,X(150 )»LT(3Q00) ^ „„ „ iOP . , _ 

common /blocki/ yb ar * ysd » mpop» spqp • kpqp# nd is t * base > i pr t 
dimension test < 150 ) 

IF ( l PR T ,E0. 1 ) WRITE ( 6 # 1 ) 

SfcSTMsO. 

SEST V s 0 . 



NSM = 0 

DO IqOO I s 1 » NT 



SV = 0 . 

SYY=0, 

DO a K = l» IPT 



t >08 P(K)=Q. 

ISTART = IPf*( I-D+l 
lST0P=lPr*I 
DC 500 J s 1 * NSP I 
CALL DATA (TEST) 

LL = 0 

DO 5 o5> K = t STAR I . iSTOp 
KK=LT(K) 

LL =LL + 1 

60^ X(LL)=TEST (KK) 

Y=0 . 

DO 510 K=l* IPT 
T = X ( K ) 

P(K)sP(K) + I 
510 Y=Y+T 

SY=SY+Y 



POO SYY=SYY+Y«Y 
XBR=SY/NSPT 

VR= ( SYY-SY*SY/NSPT )/NSPT 



SPQ=0 , 

DO 520 J S 1 * IPT 
PP=P( JJ/NSPT 



5 20 



SPQ=SPQ+PP*( 1. -PP> ' 

NSMsNSM+1 

ESTM“KP0P*XBR/1PT 

ES I Vs(NSPT»KPOP»( ( KP0P-1 , > *VR«»(KPOP- IPT ) *SPQ> ) / ( IPT# ( 1 PT-1 . 



1 ( NSP T- 1 , ) > 
SESlMsSESTM+ESfM 
SeSTVsSESTV+ESTV 



ES 1 S = Q , 

IK ( ESTV , 0 T , 
If ( ESTV r LT , 
IF ( I PR T ,EQ. 
1000 CONTINUE 



0, ) ESTS=SORT ( ESTV ) 

0. ) E8TS«-1.#SQRT(AB8C6STV)> 

1 ) WRITE <6#2> 1 ,ESTM.ES1V,ESTS 



)* 
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XhAHsSlrSlM/NSM 

XVAHsSc'STV/NSM 

«sNbPT*NSM 

|( ( M ,LT, 50 U ) XVARsXVAR*(M-X. >/M 

R t TURN 

r {IHMAT (///3bX» 22H tSTlMATb OF PARAMbTfcH///9X , 6HbAMHLL * 1 OX , 
14HMbAN,16X ,8HVARIANCfci 12X» 12HSTANDARD Ok\t//) 

FORMAT (U0.3F20.7) 

UD 




93 
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SUtlHOUTINt DATA (X> 



f; generation w item scohes and TEST Score fur hypothetical examlnee 



U 

C 



REAL mpop 

common R /blScki/ year. ysd, mpop. spop, kpop, nuisi, base, intprt 
COMMON /BLOCK2/ RNDU50) 

COMMON /BLUCK3/ P ( 1B0 ) » Q ( 1S»0 ) . NSUB 
DIMENSION X(l50) 

G fN fcR A f t TOTAL TEST SCORE 



C 



21b 

220 
2 30 



NSUBsNSUB+1 
CALL RAN UNO (Z> 

GO 10 <215,220 » 220>#-NDIST 

T£MP=Z*SPOP+MPOP 

GO 10 2 30 

TgMP = EXP( Z#YSD+YBAH ) 

Tf ( TEMP ,LT. 0, ) TEMPsO, 

Ifr ( t&MP .GT. FLOAT(KPQP) ) TtMP = KpOP 
KK * T fcMP + l # 

IF ( KK .01. KPOP ) KK=KPOP 



TSCORE =T EMp+ ( K K*RN 0 ( KK ) ) • 

IF ' TSCURb .LT. 0 ) TSCORb=0 

IF' ( T SC OR E , G I . KPOP ) TSCORE=KPOP 

If ( NDlST ,E0. 3 ) l SCORE 3 KPOP- TSCORE 



J GENERATE ITEM SCOWLS ► OR EXAMINEE 

C 

DO 240 J=1 * KPOP 
2 40 X ( J ) = 0 . 

IF ( TScURE ,E u . 0 ) GO To 300 
IF ( T SCORE .U. KPOP ) GO TO 240 
DO 242 J- 1 i KPOP 
242 *(J)=1. 

GO TO 300 
248 KOUNT=0 

f?fIxlo(J) P .1000.) .GT. IFIX(P(J>.10«0.) ) GO TO 250 

POUNTsKOUNl +1 , n 

II ( KOLJNT .GT, TSCORE ) GO TO 300 

X ( J ) =1 . 

2SQ CONT INUE 

DO 260 J=1*KP0P „ 

i r ( irixcxcj) > .eq. i ) go to 260 
KOUNT sKOUNT + 1 • _ _ 

IF ( KOUNT ,GT. TSCORE ) GO TO 300 

X( J ) =1 . 

260 CONTINUE 

300 DO 320 J=l.KPOP 

320 Q( J)*(Q( J)»<NSUB-1,)*X(J))/NSU3 
RETURN 



ENO 
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SljUROU T 1 Nt NtGHGR (K»A»B#C#HX) 



N E G A 1 l V b HV PtHGEOMbTH l C FUNCTION 



IDO 
1 1? U 



REAL n* m» mpop 

COMMON N( lbO ) # M( 1!?0 ) * LT ( 3000 ) 

COMMON /BL0CK1/ YBAR#YSD#MPOP|SPGP»KPOP» 

]F ( K .EO. 0 ) GO TO IbO 

S-KPOP 



NUIS f.BASE, INTPK 



bt 0Gl*0. 

Sl_0(i2 = 0 , 

Sl.0G3=0. 

$LUG4=0 . 

L'O 100 lsl.K 

SlOG1=SLOG1+ALOG10(S-I+1, ) 

Sl.OG2 = SLOG2 + ALOGlO(A+l-l, ) 

Si OG3=SLOC53f ALOG10<B-I + 1. ) 

SL OG4sSLOG4*ALOG10 (FLOAT ( 1 ) ) 
hX=C*lO.**(SLOGl+SLOG2-SLOG3*SLOG4> 



Rb TURN 
HX=C 
return 
eno 





91- 



buBKOUT 1 iMt ALLOU ( NT , i PT , l SA MP ) 
RANDOM ASSIGNMENT OF ITEMS TO SUBTESTS 



100 



130 

165 



170 



180 



183 

185 

150 

200 



tffl 0 



.EQ. 2 ) GO TO 200 



) 

KPUP 
GT. 0 



JJ = 1 

) JJ=KPOP 
) GO TO 170 



220 



REAL MPOP 

COMMON X(300 ) , LT (3000) _ 

COMMON /BLOCK1/ YBAR , YSD , MPOP , SPOP , KPOP » ND I ST , BASE , I NTPRT 
DIMENSION L( 150) ,KNTR<150) 

DO 100 1=1, KPOP 
KNTRC I )=0 
L ( I ) = I 
NN=NT*IPT 
IF ( I SAMP 
K = 0 

DO 150 1=1, NN 
RsKUNIF (BASE ) 

JJ=R*KPOP+l. 

IF ( JJ .LT. 1 
IF ( JJ .GT 
IF ( L ( J J ) 

GO TO 165 
LT ( I )=L< JJ) 

K = K + 1 

IF ( ISAMP 
L< JJ)=-L< JJ) 

GO TO 150 
IF ( K . LT . IPT 
i\= U 

UU 183 i 1 = 1, KPUM 
L( 1 1 ) = 1 AbS f L l 11)1 
GO 'VO 150 
L. ( JJ ) =-L ( JJ ) 

continue 

RETURN 

mmult=nn/kpop 

IF ( IK IX ( ( FLOAT ( NN )/KPOP )#10 . ) ,NE . IF IX(F LOAT (NMULT )*10 . ) ) 
K = 0 

nstup=nn-ipt 
DO 300 I =1 * NSTUP 
R=RUNIF(BASfc) 

Jj=K«KP0P+1. 

IF ( JJ .LT. 1 ) JJ=1 

KPOP ) JJ=KPOP 

GT. 0 i AND . KNTR(JJ) .LT. NMULT ) GO TO 220 



NE. 1 ) GO TO 180 



) GO TO 185 



IF < JJ *GT 
IF ( L ( J J ) 

GO TO 210 
LT ( 1, ) = L( JJ) 

K-K + l 

KNTR< JJ) =KNTR( JJ)+1 
IF ( K .LT. IPI ) GO 
K= 0 



TO 



O 

ERIC 



GOTO400 



250 



DC 230 J=l»KPOP 
L( Ji =IABS<L< J) ) 

00 TO 300 
UJJ) *-L < UO > 

C 0 N T I N U E 

DO 350 I=1#KP0P 

IF ( KNTK(l) .tQ. NMOLT ) GO TO 350 

NSTuPsNSTUP+1 

uT I NSTOP ) = 1 

CONTINUE 

RE I URN 

IF ( NN . U T . KPOP ) ISAMRaO 

1 f- ( NN . EQ . KPOP ) ISAMPal 
WRITE (6#1) ISAMP 

r\ n T 0 1 3 u 

Format <2vh tk not integer multiple of k//30h item-sampling switch 

, RESET T 0 1 5/ / ) 

END 











,’! *Vf‘, MAI 1 ,i , M*[r5;«|i-c ■’:■**>»’<! r:i * ■■."rt»i>» W t“ 
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SUBHOUTINt KANDNP (X) 

COMMON P 5aLOCKl/ YB AK . YSD .MPOP. SPOP . KPUP, ND 1ST . BASE . I NI PH T 

DIMENSION C(290).Cl(»0>.C2(8i>).C3(45).C4 60 ,CS(10> 

l. q u I VALE WCt ( V I AIL#C(200 ) ) . ( Cl (1 ) * C( 1> ) * IC2(1) »C(91> ) . 

1 ( C3 < 1 ) » C < 1 76 ) ) ,<C4(1)»C(221) ) , (C5tl), 0(281)) 

DATA Cl/ 

1 , 2 , . 2 * . 3 , . 3 , . 3 , . 3 > ,3».5» • 6 » * £ ’ 

, 6 » , 6 , . 6 » ,8, .8* , 8 » 1 . » 1 • *1*5*0. » 
o.,o.*b.,o.,o.»o.*.i**i*.i**i» 

.1. . 1 * . 1 * . 2 , . 2 « .2*. 2* .2, ,3* ,3, 



2 

3 

4 

5 

6 

7 

8 
9 

1 

2 

3 

4 

5 

6 

7 

8 
9 



,4, .4* .A, .4, .4, .4* . 5 , , 5 » • 5 » . 5 * 

. 5 * , 6 , . 7 *. 7 ,./*. 7 , . 7 * . 6 , . 8 * . 9 , 

.9, .9, .9,1. ,1. .1.1* 1.1 *1;1» I- 2 * 1>2 » 

1.2,1.3,1.3,1.4,1.4,1.5,1.6,l./*1.8».w* 

. 4 , . 4 , . 7 * • 9 , . 9 , . 9 * 1 * 1 * 1 ♦ 1 » 1 * 1 * 1; 1 7 

D i!3,i?3, 1.3, 1.3,1 .3*1*3* 1.4, 1.4, 1.6 *1*6, 

1.6, 1,6, 1.6,1 .6,1 ./, 1.7,1./, 1.8, 1.9,1. , 

1.9, 1.9*1. 9, 1.9, 1.9, 1.9, 2, ,2. *2. *2.* 

2 .* 2. ,2. *2. 1, 2.1,2. 1*2. 1, 2 . d L 6 

2. 2*2* 2*2. 2 *2‘3*2. 3, 2. 3, 2. 4, 2. 4, 2. 5*2. 6, 

.7, 1.1, 1.3, .4*1., 1.9, 1.4, .9, .8* .6* 

,5, 1,2, 1.6, 1.7, .3*1. 5,2., 1.8,2. 2, .2*^ 

*• ' ' ,946409288 , .949496939, 



. 94321650 / , 
(jA T A C 3/ 

1 .986489620, 

2 .971936598* 

3 .983249373, 

4 .990207369, 

5 .994845636, 

6 .942278196, 

7 .956691427, 

8 .9712916/8, 

9 .983065206, 
DATA C 4/ 

1 .989490775, 

. 994262546, 



.952578378, .955556/64 / 



.9613BB536* 

.974474970* 

.985020795, 

. 991260517 , 

. 995501310 , 
. 9455 72Q77 , 
.960485017 , 
.974201251* 
. 984224076, 



,964198279 , 

, 97694262 7 , 

. 9864 4 8314 , 
. 992236259, 

. 995889/39 , 
. 948551446 * 
.963804134 , 
. 976132812* 
. 966325151, 



.966788825* 

. 979212915, 
.967806989, 
, 99 3158 20 5, 

. 996268373, 
. 951165313* 
. 9665717 75 , 
. 976422883, 
.987141582, 



. 969367 756 , 
. 981233554, 
.989110415, 
. 994021949 , 
.997300203 , 
. 954986329 , 
.9689169/0 
. 9805 /9525 
.988832851 



2 

3 

4 

5 

6 

7 

8 
9 



. 990781611 , 
. 9951 10801* 



. 993063286 , 
. 996077866, 



O 

ERIC 



.991730598, 

:*42: '.994..9B6. .985, .890. ,988. .980. ’515’ 

.9 73*. 975*. 974*. 978*. 755*. 970*. 501 *.971*. 968 *.96/, 

12 . 5, 8, 2052333* 6.9186539, 20 ,, 9 .032557 9, 

4.64444 48*6.4 086308,10 . , 3 1.111111*14.^5714, 

16.666666,7.5104139,5.5743498,5.2288616,25. , 

5 . 9645244 . 4 . 3951201, 4 .9208132, 3 . 9631 7B6 * 33 . 333333 

l D s!442 7956 .3 .7 74 88 44,3 .602 0289 .4.169 0656,5 0., 

2 3 , 1592514 , 1 00 ., 3. 295642 4 , 3 , 03 24 898 , 2 . 91 4378 2 / 

5>C.N=1. 

U = RUIMlF ( BASb ) 

IF ( U . LT . .5 > SGN=~1 . 

U=RUNlF(BASb) 

IV 2*1000. 



.993813410 

.996413834, 
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1 0 

2 0 
2 1 



23 



2 4 

2 *3 
60 



5 2 



I Vl= 1 V2/10 

V a lOO.*U w »l*FLOAT ( I V 2 ) 

If ( U , (it . .79 ) GO TO 10 

X = (C< I Vl + 1 > ♦ V > »GGN 

HE I URN 

if ( U .Ob. .94 ) GO TO 20 

x - C C ( IV2-7/Q)+V)*SGN 
Kb TURN 

If- ( U .lit. VTAIL ) GO TO 30 

J = l/0 



Js.J*l 

1 f ( u , Gfc . C(j) ) GO TO 21 
J f ( u . 1.1 . C ( J*30 ) ) GO TO 23 
U=RUNI F ( BASb ) 

X s < C( J” 30 )+ ,1»U)*SGN 



HE I U H N 

U=KUN I F ( d ASb ) 

V“ RUiNl F ( d A S t ) 

Ul = AM IN1( u» V> 

U2 = A H A Xl (U , V ) 

]f ( U2 .lit. ClJ+60) ) GO 

X* (C c J-30) ♦. l*ul )*SfiN 



ro 25 



i N s»,5*(,l*Ul-,l)*C2,*C(J« , 30}*,l»Ul + .l) 

II ( (EXP(W)-1.)»C ( U*90)-U2+U1 ) 2 3 , 2 3 > 2 4 



UlsRUN IF (BASE) 

U2 = RUN l F ( BASE ) 

S = U 1 »U 1+ U2 *U ? 

IF ( S .Gfc. 1. ) GO 10 30 

T-SQRT ( ( 9. -2 .«AUOG (S ) ) /S ) 

I F ( Ul* > • LE . 3. ) GO TO 32 



X = U1«T*SG.J 
K £ T U ii N 

If c U 2* T .LI. 3. ) GO r 0 30 

X = U 2 * T * S G N 



WF turn 

END 
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r 



bllbhOUl IWt. 



AND lOGNORMAL njNCUUNb. 



A \ J b A 
1 nr 
TtST 



oniliv, l Isl ( . S IJ •< ' . O' J ! 1 Nh F U H NORMAL 

H t -; T w t* tz IS1 ADJACbNT INTfcGfeh SCORbS IS COMPljTbU HY MbANS-Gb 
ThiPF7(Hl] KOH MU LA THb ROUNUlNG VALUb IS THAT CONTINUOUS 

Such %»> u^-halk of THb ahho w.thim ths scohk . 

PQ IN T . 



A rt 0 V f THAT 



R L A L M P J P 
COMMON / ML Of- K l / 
COMMON /BLOCK?/ 
j; ! MbNS ION YdOl) 



YBAR.YSO.MPOP. SP OP, K POP. NulST.UA Sb 

RNU( 150 ) 



1 PR 1 



40 



50 

60 

C 

c 



130 

131 

135 

150 



153 

155 



[3FLT As .01 
yvar=ysd*ysu 

VP0P=SP0P*5P0P 
PI=5. 141592/ 

(i Q TO (40,50,50), NO 1ST 

CNOl-1. /SQHT(2 .*P l*VPOP ) 
CND2 = 2 .*VPOP 
GO TO 60 

CLNl=i • /SOHT ( 2 . »PI *YVAK ) 
Cl N2=2. « YVAP 

continue 



SAKEA = 0 . 

IK ( IPRT . bQ. 1 > WHlTb (6 »7> 

00 100 I=l.KPOP 

N= I -1 

DO 150 J=l*10l 



K- J~1 

X=N+K*DELTA 

GO 10 (130 .131 .131 > * NO 1ST 
Y(J)=CNDl*bXP( - ( (X-MP0P)*'*2)/CND2) 
GO TO 150 

IF < X , GT . 0. ) GO TO 135 

Y( J) =0 . 



GO TO 150 

Y(J) = (CLNl/X)ebXP(-( (ALOG(X) 
CONTI NUb ' 

IF, ( 1 PRT ,bO. 1) WRITE (6,5) 
N 1 = Y ( 1) *10000 . 
N2=Y(101)*lUOO0. 

IF ( N1 . GT . 0 . OR , N2 , GT . 



• YBAR) *•» 2 ) /C L N 2 ) 

N, ( Y< J) , J=1,101,10) 

0 ) GO TO 153 



RND < I ) =N» . 5 



GC TO 100 
AR bA =0 , 

DO 155 J=2»100 
A R b A Y ( J) 
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AKhA=UfcLlA*((V<l) +Y <101))/2.*AHfcA) 
cAkF A=SAKtA-*- A«tA 
F s U . 

i u.' 160 J =1,1 00 
K: J + l 

P = P+ ( (Y(J) + V(K) )*PtLTA/2, ) / A Kfc A 
i I ( P ,LI . . S ) GO TO 16 0 

k M) ( I ) = IM ♦ -J * U F L T A 
'.j L- I U 10U 
1 » 0 {‘ ii iM I .1 NUt: 

In R 1 lb ( 6 • 6 ) 

CAl.L t X ! T 
1(H) CGN1IMJE 

n ( I PH T .EG. U ) KfcTUKK 
WHITE (6.6) SARfeA 
WRIT f; (6,1) 

UO 2UU I r 1 » K P 0 P 
J- 1-1 

£ , 0 0 M< 1 T b (6,2) J, PNU< 1 ) . 1 



FORMAT (34H PROBLEM IN ROUND SU8K EXIT CALLED) 
F OHMAT ( IX , l 3 . 11F 8 . A ) 

FORMAT ( /H AREA =Flb.///) 



K F I U K N 

FORMAT (///2?U ROUNDING RULE 
f 6 R M A T ( 1 7 X , 1 5 , ► 1 0 . 2 , b X . lb) 



SCORE ,bX. bh ROUND, bX»bHSCC) RE/ / ) 



7 



1 

2 

INI) 





ERIC 
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